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Executive Summary 


In the last decade larger proportions of low-income, 
African American and Hispanic high school graduates 
have enrolled in postsecondary education programs. 
But their college degree completion rates have 

not kept pace. Ethnicity and income level gaps in 
bachelor’s degree attainment did not shrink at all 
between 2005 and 2015.! 


Research has shown that the number of credit- 
bearing courses that students complete successfully 
during their first few terms of college enrollment 

can predict whether or not they will earn a college 
degree. Students from low-income and under- 
represented minority backgrounds are especially 
likely to struggle with required developmental 
(remedial) and gateway (introductory) courses. When 
students drop these courses or earn a grade too low 
to count for credit toward graduation, they end up 
with fewer college credits in their initial year of college 
studies. For students from any background, those 
who do not succeed in accumulating 30 course 
credits in their first year of college are more likely to 
leave higher education without earning a degree.? 


The Bill & Melinda Gates Foundation is committed 

to increasing the number of young adults from low- 
income and under-represented minority backgrounds 
who earn a postsecondary degree or credential. The 
foundation believes that one of the levers for improving 
students’ likelinood of success in foundational college 
courses is the provision of better, more individualized 
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learning opportunities. Over the last decade, the ability 


of digital learning systems to adapt to the prior learning, 
learning preferences, and mindsets of individual learners 
has expanded considerably. The Bill & Melinda Gates 
Foundation has been pursuing the question of whether 
adaptive courseware—digital content packaged as 

an entire course for delivery over the Internet—could 
enhance student engagement and outcomes in lower- 
division college courses, especially for low-income, first- 
generation, and under-represented minority students. 
But surveys of higher education administrators and 
instructors reveal that they continue to be reluctant to 
adopt digital courseware because of concerns around 
the time requirements that implementing it imposes on 
faculty, fears about losing control over course content, 
perceived misalignment between courseware designs 
and learning science principles, and uncertainty about 
courseware’s efficacy in supporting student learning.® 


To address the limited availability of affordable, 
high-quality adaptive courseware, the foundation 
launched the Next Generation Courseware Challenge 
(NGCC) in 2014. The intent of NGCC, as explained 
in the foundation’s request for applications, was 


1 Ma, J., Pender, M., & Welch, M. (2016). Education Pays 2016: The Benefits of Higher Education for Individuals and Society. New York: The 


College Board. 


2 Attewell, P., & Monaghan, D. (2016). How many credits should an undergraduate take? Research in Higher Education, 57, 682-713. 


3 Tyton Partners. (2015; 2017). Time for Class: Lessons for the Future of Digital Learning in Higher Education. Boston: Author. 
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to encourage the development and adoption of 
courseware that would “surpass what’s currently 
available in terms of quality, price, scalability, 
learning science engineering, design excellence, 
and improvement of student outcomes.” Through a 
competitive process, seven startup, nonprofit, and 
academic organizations received three-year grants 
to develop, iterate, refine, and scale the adoption of 
adaptive courseware products for higher education: 


e Acrobatig 

° Cerego 

e CogBooks 

e Lumen Learning 

e OpenStax (Rice University) 


e Smart Sparrow 
e Stanford Open Learning Initiative (OLI) 


In addition to increasing the availability of high-quality 
courseware options, the foundation wanted to test 
whether such early-stage courseware companies 
would be able to implement and improve their products 
within a three-year timeframe using feedback from 
colleges, instructors, and students. And of course, the 
foundation wanted to test the viability of the hypothesis 
that the implementation of high-quality courseware 
could improve course outcomes for students. 


At the time the NGCC initiative was launched, a 
reasonable number of research studies had compared 
the relative efficacy of different instructional modalities 
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(online, blended, and face-to-face instruction), but 
fewer studies had evaluated the impacts of using 
adaptive digital courseware. Meta-analyses of the 
research literature generally found online learning to be 
equivalent to face-to-face instruction and blended or 
hybrid learning (a combination of significant amounts 
of both online and in-person learning) to be superior 
to both in terms of learning outcomes (Bernard et al, 
2004; Means et al., 2010). But most of the studies 

in these quantitative syntheses did not involve whole 
courses. Several studies of college courses (both 
secondary analyses of higher education data and 
experimental studies) raised concerns over whether 
online and blended courses are a good match for the 
learning needs of low-income and under-represented 
minority students.* Further, William G. Bowen, 
president emeritus of Princeton university pointed out 
that while online and blended courses might increase 
access to higher education for a diverse group of 
students and reduce the costs of providing it, this 
effort could be counterproductive if these benefits are 
obtained at the cost of increased achievement gaps 
between different income and ethnic groups.® 


The research studies that had been done at the time 
Bowen was writing his influential opinion piece did not 
involve professionally designed adaptive courseware. 
Such products were only starting to emerge in the 
higher education market, and independent tests of 
their effectiveness were yet to be done. A few years 
later, Yarnall, Means, and Wetzel conducted a study 


4 Figlio, D., Rush, M., Yin, L. (2018). Is It Live or Is It Internet? Experimental estimates of the effects of online instruction on student learning. 
Journal of Labor Economics, University of Chicago Press, 31(4), 763-784; Jaggers, S. (2011). Online learning: Does it help low-income and 
underprepared students? Retrieved from Community College Research Center, Teachers College, Columbia University; Xu, D., & Jagger, S. S. 
(2011). The effectiveness of distance education across Virginia’s community colleges: Evidence from introductory college-level math and English 


courses. Educational Evaluation and Policy Analysis, 33, 360. 


5 Bowen, W. G. (2013). Higher Education in the Digital Age. Princeton University Press. Retrieved from http:/Awww.jstor.org/stable/j.ctt24hrb1 
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of 16 early implementations of adaptive courseware.® 
That study found considerable variation in course 
outcomes for adaptive courseware compared to 
those of versions of the courses that had been used 
in the past. In launching the NGCC initiative, the 
foundation hoped to test whether newer adaptive 
learning courseware products, piloted and refined in 
the first phase of the initiative, would produce more 
consistently positive course outcomes than those 
found in the earlier adaptive courseware research. 


SRI International, a nonprofit research and 
development organization, was retained to evaluate 
the NGCC initiative as a whole and to assist the 
grantees in conducting pilot and impact studies 

for the courseware they developed and marketed 
with NGCC funding. SRI maintained term-by-term 
records of student enrollment in course sections 
using each NGCC product, collected information on 
implementation challenges and supports through 
campus visits to pilot sites, advised grantee staff 
on impact study design, collected instructor 
perceptions through surveys, analyzed student-level 
impact data provided by the grantee organizations 
or their higher education partners, and performed 
cost effectiveness analyses for a subset of the 
courseware implementations. 
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Major Findings 
Scaling 


The NGCC courseware was used broadly and 

in a diverse set of higher education institutions. 

The seven NGCC grantees submitted data to SRI 
showing that their courseware was used by over 
138,000 undergraduates and over 1,000 instructors 
in 449 different higher education institutions during 
the three-year period from 2015 to 2017. Over 40% 
of the institutions adopting the courseware were 
public two-year colleges with the remainder being a 
mix of public and private, traditional or online, four- 
year institutions. 


Grantees employed different strategies for scaling 
adoption of their courseware. Some relied mostly 

on adoptions by individual faculty members, which 
aligns with the usual practice at most higher 
education institutions.’ Other grantees pursued 
institutional adoptions by whole departments or 
programs. The individual adoption strategy 
resulted in higher numbers of adoptions in the 
first portion of the grant period but by the end 
of three years, institutional adoptions reached 
the same cumulative total in terms of number of 
students reached. This finding is important because 
institutional adoptions are difficult for new technology 
companies to negotiate but will be necessary if 
colleges and universities are to leverage adaptive 
courseware at scale to improve course success 

rates and persistence for low-income and under- 
represented minority students. 


6 Yarnall, L., Means, B., & Wetzel, T. (2016). Lessons Learned from Early Implementations of Adaptive Courseware. Menlo Park: SRI International. 


Also funded by the Bill & Melinda Gates Foundation. 
7 Tyton Partners. (2017). 
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Perceptions of Courseware Strengths 
and Weaknesses 


Insights from Pilot Testing. When NGCC grantees 
rolled out their new courseware products to pilot 
campuses, they received feedback from students 
and instructors. In addition to this feedback, SRI 
researchers visited a number of pilot sites in order to 
observe courseware implementation and interview 
instructors, and researcher’s observations were 
shared with the developers. We found that both 
instructors and students involved in the pilot 
testing reported liking the interactive elements 
of next-generation courseware and the adaptive 
formative assessments embedded within the 
courseware. At the same time, pilot experiences 
revealed a number of improvements that instructors 
and students wanted to see the in the courseware: 


e In some cases, instructors and/or students 
wanted more guidance on how to use the 
courseware before starting to work with it. 


e Some instructors wanted the capability to 
modify the courseware to better fit the learning 
objectives they had for the course or to adjust 
the software’s mastery criterion to one they 
deemed more appropriate for their students. 


e Using courseware products on mobile devices was 
found to place some limits on use of interactive 
media and ability to work on complex problems. 


e Many students did not use courseware- 
embedded review functions as intended. This 
may have been because the software did not 
force them to review material they had not 
mastered, and students prioritized getting 
through the courseware module as quickly as 
possible (rather than maximizing their learning 
long term). 
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e Instructors did not always leverage all of the 
courseware’s capabilities, suggesting the need 
for more instructor training and support. 


As the foundation had intended, NGCC grantees 
refined their products in response to this kind 
of feedback from their early users. However, 

in some cases there was not time to make the 
refinements in products prior to initiating the 
courseware impact studies. 


After the initiative’s pilot phase, instructor perceptions 
surrounding the courseware and their experiences 
implementing it were gathered through a survey that 
SRI conducted in fall 2016 with additional survey 
waves for new instructors in spring 2017 and fall 
2017 in cases where the instructor’s class data were 
expected to be part of an impact study. The survey 
response rate was a healthy 78%, with 550 NGCC 
instructors completing the survey. 


Instructor Preparation for Teaching with 
Courseware. The instructor survey provided a 
profile of the course instructors who used NGCC 
products. Of these instructors, 73% were teaching 
full time, and 52% were tenured or in a tenure-track 
position. Survey responses indicated further that the 
majority of instructors (70%) had volunteered to use 
the courseware. Their main motivations for doing 

so were the beliefs that it would increase students’ 
engagement with course content and that it would 
enable them to do more individualizing of instruction. 
Two-thirds of the NGCC instructors had taught an 
online or blended learning class before, and a similar 
percentage (69%) felt that they had been adequately 
or well prepared to implement the courseware. 
Instructors did not spend an inordinate amount of 
time preparing to teach with the courseware. They 


ES-4 


reported spending an average of 8.37 hours learning 
to use the courseware and 16.24 hours integrating 
the courseware into their courses. 


Instructor Perceptions of Courseware Impacts. 
After using the courseware, a majority of NGCC 
instructors felt that it had improved teaching and 
learning in their class: 


© 63% agreed or strongly agreed that the 
courseware supported students in deeper 
learning, 


© 68% agreed it increased student engagement 
with course content, 


e 57% thought it enabled better monitoring of 
individual student progress, 


e 58% said it resulted in better tracking of whole- 
class progress, and 


e 52% agreed that content was presented more 
effectively with the courseware. 


Most instructors agreed that using the courseware 
provided the following benefits for their students’ 
learning: 


e better understanding and remembering of course 
content (66%), 


e staying more engaged with the course overall 
(64%), and 


e better analysis and application of course content 
(62%). 


Overall, the great majority of NGCC instructors 
reported being either moderately or highly 
satisfied with the courseware (88%). These 
very positive instructor responses were especially 
noteworthy in light of prior reports in the literature. 
In addition to the overall dim views of courseware 


8 Tyton Partners, 2015; 2017; Yarnall, Means, & Wetzel, (2016). 
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on the part of higher education administrators and 
faculty documented by Tyton Partners, the earlier SRI 
evaluation of adaptive courseware found that fewer 
than half of the instructors surveyed in that study 
planned to use the adaptive courseware they had 
tried out in the future.® In contrast, 80% of NGCC 
instructor respondents said they planned to 

use the NGCC product in their course in the 
future. In terms of the faculty negative perceptions 
of courseware in general found by Tyton Partners, 
The NGCC products appear to have surmounted 
some of the specific courseware drawbacks cited by 
faculty surveyed by Tyton Partners: They were flexible 
enough to accommodate many different models 

of blended and online learning, thereby mitigating 
instructor concerns about loss of control over one’s 
course and imposition of an inappropriate pedagogy. 
NGCC instructors’ perception of positive impacts on 
student learning and their plan to use the courseware 
in the future suggest further that skepticism about 
courseware effectiveness and the burden that 
implementing it places on faculty were not an issue 
for the instructors who tried out NGCC products. 


Yet despite NGCC instructors’ quite favorable 
responses to the courseware products they had 
used and their intention to Keep using them, they 
were not as enthusiastic as one might expect in their 
responses to a question about whether they would 
recommend the courseware they had used to a friend 
or colleague teaching the same course (average 
response 6.52 on a scale from O = Extremely Unlikely 
to 10 = Extremely Likely). To inform efforts to build 

a market for adaptive courseware, future research 
should probe for reasons why instructors are not 
more likely to recommend a courseware product they 
like to others in their field. 


ES-5 


Effects of Courseware Implementations 
on Students’ Course Grades 


NGCC grantees and the higher education 
institutions working with them submitted courseware 
implementation datasets to SRI. Each dataset 
contained student-level variables (gender, race/ 
ethnicity, Pell status, age, first-generation college 
goer, full- or part-time enrollment, prior attempts in 
this course), a prior achievement measure (typically, 
college grade point average), and a common 
outcome measure (almost always course grade) for 
students in “treatment” course sections using the 
NGCC courseware and comparison or “business- 
as-usual” sections of the same course that did not 
use the courseware. Analysts identified the subset 
of these datasets that met predetermined criteria 

for inclusion in impact analyses (i.e., having enough 
students in each condition and having students in the 
two conditions who were reasonably similar to each 
other before the course began). Twenty-eight of the 
NGCC datasets met both of these criteria. 


After applying statistical modeling to control for 
differences between treatment and comparison 
conditions in terms of student characteristics and 
prior achievement, analysts found that NGCC 
courseware effects on student grades varied 
widely, as shown in Figure ES-1. Students using the 
NGCC courseware earned significantly higher grades 
than those in the comparison or business-asusual 
versions of the course in 10 of the 28 impact studies 
(those with boldface labels in the figure).° 
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In seven of these cases, the effects of courseware 
implementation were sizable enough to have practical 
consequences. With positive effect sizes of .30 or 
greater, '° this quarter of the NGCC courseware 
implementations had impacts equivalent 

to moving an average student at the 50th 
percentile in course performance to the 62nd 
percentile or higher. On the other hand, 4 of the 
28 NGCC course implementation impact studies 
found a statistically significant negative effect. In 
the other half of the datasets, student grades in the 
two conditions were statistically equivalent after 
controlling for differences in student characteristics, 
such as full-time enrollment and prior grade point 
average. 


When positive, negative, and no-difference results 
across the different products, institutions, and 
courses were averaged together, the mean NGCC 
courseware impact on grades was small 

but positive and statistically significant. The 
average NGCC courseware effect size of +0.09 was 
equivalent to moving an average student at the 50th 
percentile in the course (as it was taught in the past) 
to the 54th percentile through the implementation of 
NGCC courseware. The size of this average impact 
was similar to that found by Yarnall, Means, and 
Wetzel for 12 implementations of earlier versions of 
adaptive courseware. "! 


To address the foundation’s goal of improving course 
success and graduation rates for low-income and 

under-represented minority undergraduate students, 
we next performed separate analyses of courseware 


9 The criterion used for statistical significance was the customary p < .05, meaning that a difference as large or larger than the observed 
difference would be found merely by chance in fewer than 1 out of 20 repetitions of the same study if the populations of treatment and 


comparison students did not differ in course performance. 


10 An effect size is the difference between the treatment and the comparison group average expressed in standard deviation units. 


11 Yarnall, Means, and Wetzel (2016). 
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impacts for these student groups for those data sets 
with adequate numbers of low-income and under- 
represented minority students in each condition (7 
data sets for low-income students and 10 for under- 
represented minority students). 


On average, low-income students using NGCC 
courseware had grades that were +0.10 standard 
deviation higher than those of low-income students 

in the business-as-usual versions of the course, an 
average effect size that was almost identical to that for 
students overall. Although this difference in grades, like 
that for students overall, is small in magnitude, it does 
show that low-income students’ learning wasn’t 
undermined by using courseware. 


The results for under-represented minority students 
(African American or Latino) were even more 
encouraging. The average courseware impact on 
under-represented minority students’ grades was an 
improvement of +0.16 standard deviation units, a 
statistically significant effect. An effect size of 0.16 is 
equivalent to an under-represented minority student 
at the 50th-percentile student in the traditional 
course moving to the 56th-percentile through the use 
of the NGCC courseware. 


Figure ES-2 shows the distribution of impact findings 
(positive, neutral, or negative) for under-represented 
minority and low-income students in the individual 
courseware implementations in this analysis. Half 

of the impact estimates for under-represented 
minority students were significantly positive; 
and each of these studies found an impact 
greater than +0.20 standard deviations. Only one 
courseware implementation study found a statistically 
significant negative impact for underrepresented 
minority students. 
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Stronger evidence of the impact of new course 
designs could be amassed if higher education 
institutions set up experimental designs with random 
assignment to different versions of the course. But 
even in the absence of a large number of rigorously 
designed studies, the preponderance of evidence 
from the 28 NGCC controlled comparison studies 
addresses one of the concerns educators express 
about adaptive courseware: Students learn at 
least as well in courses modified to incorporate 
adaptive courseware as they do in prior versions 
of the same course; nor does courseware 
exacerbate achievement gaps for low-income or 
under-represented minority students. 


Still, it is important to move beyond average effects 
to try to understand (1) when and under what 
circumstances implementing adaptive courseware 
will actually improve student outcomes and (2) how 
to adapt courseware products and implementation 
practices to obtain more consistently positive 
learning outcomes. 


This level of variability in the impact estimates 
overall and for student subgroups suggests 
that improving student outcomes is not just a 
matter of finding the right courseware product. 
Despite the design qualities built into the NGCC 
products, those with larger numbers of impact 
studies (Lumen and Smart Sparrow) found positive 
impact in some cases but not in others. The way 

in which courseware is implemented—the amount 
of support instructors receive for using it, the role 
that instructors give the digital learning resources 

in their course, the fit between courseware content 
and the assessments that grades are based on, and 
the appropriateness of courseware content for the 
particular set of students in a course section—may all 
influence student outcomes. 
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Figure ES-1. NGCC courseware impact estimates for 28 implementations 


Grantee/Institution 
Acrobatiq/WGU 
Cerego/Columbia 
CogBooks/ASU 
CogBooks/SouthernU and A&M 
CogBooks/SouthernU and A&M 
Lumen/Broward 

Lumen/Broward 

Lumen/Broward 
Lumen/Broward 
Lumen/Glendale 
Lumen/Glendale 
Lumen/Pittsburg State 
Lumen/Pittsburg State 
Lumen/Pittsburg State 
Lumen/Tidewater 
Lumen/Tidewater 
Lumen/Tidewater 

Lumen/UMUC 

Lumen/UMUC 

Lumen/UMUC 

Lumen/UMUC 

Smart Sparrow/ASU 

Smart Sparrow/Central Piedmont 
Smart Sparrow/Miami-Dade 
Smart Sparrow/Mohave 

Smart Sparrow/National Louis U 
Stanford-OLI/Canyons 
Stanford-OLI/UMUC 


Course 

Introduction to Psychology 
Macroeconomics 

The Living World 

General Biology 

General Biology 

Principles of Marketing 
Introduction to Business 
Principles of Macroeconomics 
Principles of Microeconomics 


Introduction to Business 
Macroeconomics Principles 
Introduction to Business 
Introduction to Microeconomics 
Principles of Marketing 
Introduction to Business 
Principles of Macroeconomics 
Principles of Microeconomics 
Marketing Principles 

Introduction to Business & Mgmt. 
Principles of Macroeconomics 
Principles of Microeconomics 
The Living World 

Principles of Biology 

General Education Biology 
Biology Concepts 

Survey of Bioscience 
Introduction to Statistics 
Introduction to Statistics 


Total 
-1.00 1.00 
_ Favors Favors 
Business-as-Usual Courseware 


The square for each courseware implementation represents the estimated impact (effect size). The size of the square represents the weighting of 
the study in the meta-analysis. The length of the horizontal line through the square represents the confidence interval around the impact estimate. 
The longer the line, the more uncertainty there is around the true impact of the courseware implementation. The diamond at the bottom of the graph 
represents the average effect for the 28 courseware implementations. Its width represents the 95% confidence interval for the average impact. 


Squares to the right of the boldface vertical line represent studies in which students in the courseware sections outperformed students in the business- 
as-usual sections. Squares to the left of the boldface vertical line represent studies in which students in the business-as-usual courseware sections 
outperformed students in the courseware sections. Only those courseware implementations with boldface labels are statistically significant (i.e., we 
can be confident that the difference in average grades for students in courseware and business-as-usual course sections did not occur by chance). 
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Figure ES-2. Percent of studies with various impacts on grades for low-income and 


under-represented minority students 


Low-income Students 


od 


sj Negative Impact 


a No Impact 


NGCC impact estimates were examined further 

to glean insights into the conditions and practices 
associated with more and less effective courseware 
implementations. These analyses were strictly 
exploratory and should be interpreted as such 
because many features of the courseware 
implementations were highly correlated with each 
other, making it difficult to ascertain what caused 
the patterns observed in the impact data. With this 
proviso, we note that across the NGCC products and 
institutions: 


e The average courseware impact 
estimate was significantly positive for 
implementations in four-year colleges 
and statistically insignificant in two- 
year colleges. Nine of 16 courseware 
implementation studies conducted in four-year 
colleges (56%) found significantly positive 


12 Yarnall, Means, and Wetzel. (2016). 
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Under-represented minority students 


| Positive Impact 


impacts. Only 1 of the 12 studies conducted 

at two-year colleges produced significantly 
positive impacts, suggesting that more capacity 
building and research into implementation 
strategies need to be done in those settings. 
The disappointing impact findings for two- 

year colleges is somewhat surprising in light of 
previous findings that students and instructors at 
community colleges tend to have more favorable 
perceptions of adaptive courseware than do their 
counterparts at four-year institutions. '7 


On average, the implementations of 
courseware in biology, psychology, and 
math/statistics classes led to higher 
student course grades than those earned 

in business-as-usual sections of the same 
courses. In contrast, on average the courseware 
impact appeared to be insignificant for business 
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administration and economics classes (which 
were the subject area for two grantee products 
with impact data). To some extent the subject 
area findings may represent product rather than 
subject matter differences, but the pattern of 
stronger results for science and math courses is 
congruent with a nonsignificant trend found by 
Yarnall, Means, and Wetzel and with previously 
reported research indicating that digital learning 
has more positive impacts in STEM courses than 
in other subjects. '% 


The average courseware impact was 
significantly positive for implementations 

in which a blended or online version of the 
NGCC course was compared to a face-to- 
face version of the course, but not when an 
online course using NGCC courseware was 
compared to another online version of the 
course. This is not to say that online courses 
using NGCC courseware were ineffective, but 
simply to note that on average they were not 
significantly more effective than other online 
versions of the same course. There were some 
notable exceptions to this pattern, especially, the 
Western Governors University online psychology 
course using Acrobatiq courseware. 


Impact studies in which courseware-using 
sections had instructors who were full-time 
teachers had significantly positive impacts 
while those with part-time instructors did 
not. Those impact studies where instructors 
using the courseware were all or mostly 
part-time employees were for the most part 
contrasting online sections using an NGCC 
product with online sections without the product. 
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Courseware impacts were not related to whether 
sections were taught by regular faculty (tenured 
or tenure track) or by instructors or adjuncts. 


e Implementations in courses where 
instructors taught more than 50 students 
in the regular (comparison) classes led to 
more positive courseware impacts than 
those where the class sections were smaller. 
This data pattern has not been reported earlier 
but makes sense given the limited amount of 
personal attention an instructor can give to 
students in a large, conventional class. 


Cost Effectiveness of NGCC 
Courseware Implementations 


To make good decisions about implementing 
particular instructional interventions, education 
decisionmakers need to understand the likely 

costs as well as the likely benefits. For courseware 
implementations that were part of impact studies, the 
SRI evaluation team sought information concerning 
the cost ingredients for both courseware and 
comparison course sections. These ingredients 
included instructor time devoted to various activities 
such as course planning and grading as well as time 
spent with students and costs of facilities, textbooks, 
software, and so on. Collecting this kind of detailed 
cost data is labor intensive, both for researchers and 
for the administrators and practitioners involved in 
supplying the information. SRI was able to obtain 
sufficient cost information to support a quantitative 
analysis of costs for nine of the impact studies. 


13 Vo, M. H., Zhu, C., and Diep, A. N. (2017). The effect of blended learning on student performance at course-level in higher education: A meta- 


analysis. Studies in Educational Evaluation, 53, 17-28. 
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Analysis of these data employed a cost ingredients 
approach as described by Henry Levin and 
colleagues." Initial startup costs for courseware 
implementation were spread across the number of 
terms a college anticipated using the product and 
included the value of instructor time, regardless of 
whether or not instructors were compensated for 
that extra time. In 8 of the 9 cost analyses for 
NGCC courseware implementations, per-student 
costs were lower for the courseware-using 
version of the course than for business-as- 
usual. Importantly, these cost savings were not 
achieved by reducing instructor time with students 
nor by compromising student learning. In 7 of the 8 
cases where money was saved, student achievement 
actually went up compared to the comparison 
condition without courseware. 


For the nine NGCC cases in our cost-effectiveness 
analysis, use of NGCC courseware moved the 
average student’s performance from the 50th to 
the 58th percentile while saving approximately 
$105 per student. The cost savings found for these 
NGCC courseware implementations stemmed primarily 
from avoidance of textbook costs for students. 


The NGCC data present a more positive picture of cost 
effectiveness than did the cost analyses conducted as 
part of the earlier SRI study of adaptive courseware 
implementations, in which only a single courseware 
implementation out of 16 achieved both better student 
outcomes and cost reductions.'® Nevertheless, it 
should be noted that the findings described above 
represent total costs. If only costs to the higher 
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education institution are considered, implementation 

of courseware in these nine cases entailed an average 
additional cost of $107 per student. This additional 
cost to the institution might be expected to decrease 
with subsequent iterations of courseware use, aS was 
found when Yarnall, Means, and Wetzel looked at 
costs separately for three successive terms of adaptive 
courseware implementation. '® 


The Next Generation Courseware Challenge cost 
effectiveness findings come from a small number 
of cases that are not necessarily representative of 
the courseware experiences in higher education 
as a whole. Providing detailed cost data requires 
significant effort on the part of college staff, and 
institutions willing to make the effort tended to be 
those that had enjoyed more success with their 
courseware. Nevertheless, these early cost 
effectiveness findings suggest that although 
courseware is no silver bullet, it is possible 
to implement it in ways that permit progress 
on both quality and cost. Having proof that it is 
possible, the next step is understanding how best 
to implement courseware with different kinds of 
learners and to facilitate learning of a broader array 
of academic content in more diverse settings with 
similar cost effectiveness. 


14 Levin, H. M., McEwan, P.J., Belfield, C., Bowden, A. B., and Shand, R. (2017). Economic Evaluation in Education: Cost-Effectiveness and 


Benefit-Cost Analysis (3rd ed.). Thousand Oaks, CA: Sage Publications. 


15 Yarnall, Means, and Wetzel. (2016). 
16 Yarnall, Means, and Wetzel. (2016). 
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Implications for Future Initiatives 


The Next Generation Courseware Challenge 
generated both a great deal of data and valuable 
experiences for colleges, educational technology 
developers, researchers, and philanthropists with 
respect to the issues surrounding the implementation 
of digital courseware as part of a broader effort to 
close college attainment gaps. 


The NGCC experience suggests that philanthropic 
backers of postsecondary courseware would benefit 
from reconceptualizing what they mean by scale. 
Market analyses emphasize the number of product 
users, but product use is not really the goal for 
higher education institutions and philanthropies. We 
should consider the multiple dimensions of scaling, 
which include the degree of penetration within an 
institution and the extent to which the institution has 
assumed ownership for the intervention as well as 
the number of institutions involved.'” Moreover, we 
need to keep in mind the fact that what we need to 
scale is not courseware use per se but practices 
that leverage courseware in ways that improve 
student outcomes. We must continue to measure 
courseware impacts to identify cases where the 
implementation has been successful, but we then 
need to examine those cases in detail to identify and 
describe the policies, structures, and practices that 
enabled that success. 


To inform further work on closing gaps in course 
success rates, it would be helpful to conduct 
in-depth studies of the study behaviors and 
experiences of students from low-income and 
under-represented backgrounds in gateway 
courses with and without adaptive courseware. 
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Such studies could inform the product features and 
implementation of learning technologies so that the 
field can learn how to use adaptive courseware in 
ways that benefit low-income, first-generation, and 
under-represented minority students. 


Similarly, the field would benefit from a 

better understanding of the needs of two- 

year college instructors with respect to 
courseware implementation and of best 
support practices within community colleges 
that are implementing teaching and learning 
improvement efforts well. The low number of 
community college courseware implementations 

with positive impacts on student course outcomes 
suggests that these institutions have not developed 
some of the organizational capacities and processes 
used in four-year colleges. At the same time, it 
should be recognized that community colleges 

are serving a different population and often do so 
primarily with part-time instructors. There is a need to 
develop models for growing the capacity to improve 
teaching and learning within community colleges and 
to document “proof of concept” cases from which 
other community colleges can learn. 


The NGCC experience has implications for 
courseware market studies as well. Efforts to scale 
courseware through institutional adoptions may 
sacrifice user numbers in the short term, but have 
advantages in the longer term. NGCC scaling 

data suggest that it may take three years for the 
institutional scaling strategy to attain parity with the 
direct-to-faculty strategy in terms of user numbers. 
But what these numbers do not reveal is the greater 
capacity for department-wide course redesign and 
for measuring courseware impacts in institutions that 


17 Coburn, C. (2008). Rethinking scale: Moving beyond numbers to deep and lasting change. Educational Researcher, 32(6), 3-12 
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are undergoing wholesale adoption of courseware. 
To the extent that courseware is intended to 

be a stimulus for significant changes in college 
programs and pathways, institutional adoptions 
(at least program or department-wide) will 

be necessary. At the same time, it should be 
acknowledged that institutional-level courseware 
implementations are much more challenging. They 
require involvement and agreement among many 
more players and mean that the instructors who will 
be using the courseware cannot be limited to those 
who are enthusiastic about doing so. 


Implementing courseware at scale in ways that 
enhance student outcomes at the institutional 
level is the next big challenge, and it appears 

to be one that will be particularly perplexing for 
community colleges. To have a practical benefit 

for students and higher education institutions, 
courseware needs to be implemented—and 
implemented well—within a critical mass of the 
college’s courses. This need suggests that the field 
would benefit from knowledge about how a college 
or university can move beyond working with those 
instructors who are early adopters and still get good 
results for students. Close examination of successful 
efforts to do this would be extremely helpful as we 
try to make progress toward achieving consistently 
positive outcomes at scale. An evidence-driven 
deeper dive into instructor quality, instructional 
practices, student and instructor supports, and 


students’ own behaviors and perspectives is needed. 


More direct engagement between researchers 
and higher education institutions appears 
necessary both to improve the quality of 
impact evidence and to enhance organizational 
capacity. In the course of conducting the NGCC 
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and prior postsecondary learning technology 
evaluations, we have found that the majority of higher 
education institutions do not measure the learning 
impacts or per-student costs of incorporating 
learning technology into their courses. Nor do 
learning technology developers typically conduct 
rigorous studies of the impacts their products 

have in learning. The NGCC grantees should be 
commended for their willingness to submit their early 
products to evaluation studies they knew would 

be made public. Lumen Learning was particularly 
notable in this regard, submitting 16 of the 28 

data sets with adequate numbers of comparable 
students in treatment and comparison conditions 

to enable estimating the impact of the courseware 
implementation. But the field should not rely on 
learning technology product developers and vendors 
to perform needed studies of impact. Ideally, colleges 
and universities themselves would employ evidence- 
based decision-making and iterative improvement 
processes as they embrace digital learning. 

Getting more consistent, credible impact evidence 
for courseware will require having researchers, 
instructional designers, and faculty collaborate on 
redesigning courses to foster active learning and on 
articulating and measuring the impact of redesigned 
courses on student outcomes and costs. 


Courseware impact studies should expand 
their focus to encompass longer-term student 
outcomes, including persistence and credit 
accumulation over time, as well as course 
grades and credits. The foundation has amassed 
course outcome data for scores of courseware 
implementations over the last eight years. While 
useful, this database by itself does not answer 
questions about progress against the foundation’s 
goal of increasing the proportion of low-income 
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and under-represented students who attain 

college degrees or other marketable credentials. 

The foundation has taken the important step of 
sponsoring a framework of near, medium, and 
long-term key performance indicators at the 
institutional level. The next step in understanding 
how courseware use can impact these aggregate 
performance indicators will be longitudinal studies 

of student cohorts to address questions about how 
students who experienced well-implemented adaptive 
courseware in their gateway courses perform relative 
to other students in terms of credit accumulation, 
retention, grade point average, and success in 
subsequent courses in their major or pathway. Only 
by conducting well-controlled longitudinal studies 

of student outcomes can we understand whether 

or not the investment in digital learning is having 

the intended long-term payoff that is necessary for 
improving students’ lives. 
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Introduction 


The Next Generation Courseware Challenge (NGCC) 
of the Bill & Melinda Gates Foundation is part of 

the foundation’s strategy to increase the number of 
young adults from low-income and under-represented 
minority groups who earn a postsecondary degree 

or credential. The reasoning behind the initiative 

is that higher-quality, more personalized learning 
opportunities in a student’s initial college courses 
should result in higher course completion rates, with 
positive impacts on credit accumulation and hence 
college completion. The foundation wanted to explore 
the question of whether the digital learning systems 
that are becoming increasingly adaptive and capable 
of providing an array of learning activities could 
produce higher success rates for students in general, 
and for low-income and under-represented minority 
students in particular, in entry-level college courses. 


To address both the supply and the demand side 
of the digital courseware market, the Bill & Melinda 
Gates Foundation launched the NGCC in 2014. 


Definition of Digital Courseware 
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The foundation distributed its RFI-RFP for the Next 
Generation Courseware Challenge to 109 innovative 


organizations, including digital learning providers, 
publishers, media organizations and postsecondary 
institutions in May 2014. Offerors were asked to 
describe how they would take on the challenge of 
designing, developing, marketing, and delivering 
exemplary courseware for undergraduates in core 
general education courses. 


In describing what the foundation was looking for in 
“next-generation” courseware, the NGCC request for 
information and proposals (RFI-RFP) explained that 
such courseware would “surpass what’s currently 
available in terms of quality, price, scalability, 


The foundation has elaborated on Tyton Partner’s (2014) definition of courseware. This definition, below, 
explicitly excludes learning management systems because they are not course content per se. 


Courseware is “digital content, assessments and interactives, delivered over the Internet, which are packaged and 
distributed as a complete course (e.g., not a partial course, unstructured learning objects, or an optional course 
supplement). High-quality courseware includes multiple components, including expert-sourced learning outcomes, 
modular and extensible interactive learning modules and curricula architected and designed for continuous 
improvement and learner feedback, integrated formative assessments and course summative assessments, 
instructional resources and help modules supporting student and faculty use and adoption of the courseware.” 


--- Bill & Melinda Gates Foundation, NGCC RFI-RFP, 2014, p. 3. 
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learning science engineering, design excellence, and 
improvement of student outcomes.” 


The 15 attributes of high-quality next-generation 
courseware identified by the foundation are shown in 
the sidebar. 


Sixty organizations submitted preliminary NGCC 
proposals in response to the RFI-RFP, and the 
foundation selected 18 of these as finalists. The 
finalists then prepared more detailed proposals and 
gave an in-person presentation to a panel of five 
foundation staff and five external advisors at the 
foundation’s offices in Seattle. Using scoresheets 
from these panelists, the foundation selected seven 
organizations to receive three-year NGCC grant 
awards of $1-4.5 million starting in January 2015: 


e Acrobatig 

° Cerego 

e CogBooks 

e Lumen Learning 

e OpenStax (Rice University) 

e Smart Sparrow 

e Stanford Open Learning Initiative (OLI) 
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Attributes of High-Quality, 
Next-Generation Courseware 


1. Includes student-facing digital content for a 
complete course 


2. Designed for online and/or blended learning 
scenarios 


bo 


. Based on cognitive and learning science 
research 


. Designed to stimulate effective instructional 
practice and pedagogy 


uw 


.Provides a user experience that sets a 
new bar for ease of use, elegance, user 
engagement, and support for learner and 
instructor insights 


i) 


. Powered by rigorous and defensible 
articulation of learning outcomes, driving 
continuous improvement and refinement of 
both the learning content and assessments 


=~ 


Integrates dynamic and creative embedded 
assessments to promote mastery as well 
as summative assessments aligned to the 
course subject area 


. Permits faculty to customize, align, add 
or subtract content and instructional 
experiences for their specific educational 
context and learners 


so 


Reflects advances in adaptive learning to 
scaffold and accelerate a learner’s mastery 
and progression 


10. Conforms to best-in-class technologies 
and interoperability standards, learning 
accessibility standards, and web design 


11. Exceeds field-recognized course design 
rubrics such as Quality Matters and ACE 
CREDIT Course Review 


12. Optimized for multiple delivery modes 
including mobile, tablet, and desktop 


13. Provides best-in-class faculty and learner 
real-time assistance, support, and help 


14. Enables communities and groups of 
learners to support and assess each other 


15. Optimized for machine learning or data 
mining to generate learner and user 
analytics 


The Foundation’s Strategy for 
Supporting Development and 
Spread of Next-Generation 
Courseware 


At the time the NGCC initiative was launched, 

a reasonable number of research studies had 
compared the relative efficacy of different 
instructional modalities (online, blended, and face- 
to-face instruction), but fewer studies had evaluated 
the impacts of using adaptive digital courseware. 
Meta-analyses of the research literature generally 
found online learning to be equivalent to face-to- 
face instruction and blended or hybrid learning (a 
combination of significant amounts of both online and 
in-person learning) to be superior to both in terms of 
learning outcomes (Bernard et al, 2004; Means et al., 
2010). But most of the studies in these quantitative 
syntheses did not involve whole courses. Several 
studies of college courses (both secondary analyses 
of higher education data and experimental studies) 
raised concerns over whether online and blended 
courses are a good match for the learning needs of 
low-income and under-represented minority students 
(Figlio, Rush, & Yin, 2010; Jaggers, 2011; Xu & 
Jagger, 2011). Further, Bowen (2013) pointed out 
that while online and blended courses might increase 
access to higher education for a diverse group of 
students and reduce the costs of providing it, this 
effort could be counterproductive if these benefits are 
obtained at the cost of increased achievement gaps 
between different income and ethnic groups. 
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The research studies that had been done at the 
time Bowen was writing his influential opinion piece 
did not involve professionally designed adaptive 
courseware. Such products were only starting 

to emerge in the higher education market, and 
independent tests of their effectiveness were yet 

to be done. A few years later, Yarnall, Means, 

and Wetzell (2016) conducted a study of 16 early 
implementations of adaptive courseware.'® That study 
found considerable variation in course outcomes for 
adaptive courseware compared to those of versions 
of the courses that had been used in the past. In 
launching the NGCC initiative, the foundation hoped 
to test whether newer adaptive learning courseware 
products, piloted and refined in the first phase of the 
initiative, could improve student learning outcomes 
at scale, especially among low income and under- 
represented minority students. 


Terms of the NGCC grants required not only 
developing exemplary next-generation courseware 
for high-enrollment, lower-division college courses 
but also enlisting a high number of colleges and 
universities to use these products and collecting the 
data needed to (1) provide feedback on the usability 
and quality of the products in the initiative’s first 
phase and then (2) to ascertain the impacts of the 
courseware on student learning and the attainment 
of course credit. The foundation was laying out a 
major challenge by setting these multiple goals, but 
the organizations invited to bid already had digital 
learning products, and the assumption was that 
they would be extending and refining those products 
rather than starting from scratch. The key premise 
linking these activities to the foundation’s goal is that 
courseware incorporating the next-generation quality 


18 Also funded by the Bill & Melinda Gates Foundation and available at httos://www.sri.com/sites/default/files/brochures/almap_final_report.pdf. 
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features would improve outcomes for low-income 
and disadvantaged students receiving access to it. 


Another challenge implicit in the foundation’s RFI- 
RFP was the development of new business models 
because the foundation required grant applicants to 
explain how the courseware they developed under 
the grant would be made available to those learners 
most in need at low or no cost, and also to explain 
how their organization would be sustained financially 
after grant funding ended. 


To support the grantees’ Next Generation 
Courseware Challenge work, the foundation engaged 
the services of several additional organizations. 

SRI International was tasked with evaluating the 
NGCC portfolio as a whole and with helping each 
of the grantees design and execute pilot and 
impact studies. The consulting firm Intentional 
Futures was engaged to help grantees incorporate 
user experience perspectives into their design 

and development work and to provide technical 
assistance around marketing strategies. Tyton 
Partners was contracted to develop a framework 
for assessing courseware quality and to conduct a 
follow-up survey of college faculty and administrators 
to capture their satisfaction and concerns around 
courseware adoption. Subsequently, EdSurge 

was engaged to develop an online digital learning 
community and dissemination portal for information 
on postsecondary digital learning. Since then, 
additional organizations have joined the broader 
collective effort to increase postsecondary success 
through the Every Learner Everywhere network. 
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The NGCC grant terms required grantee 
organizations to carry out pilot testing in the first 
phase of their grant and subsequently to work 

with SRI to design and implement data collections 
during later courseware implementations that could 
be used to estimate the impacts of courseware 
usage on student course completion and learning 
outcomes. The NGCC research and evaluation 
activities were based on the foundation’s Courseware 
R&D Evaluation Framework (Means, Peters, & 
Zheng, 2014). That framework posits four goals for 
evaluation support: 


e Support design and development of potentially 
powerful courseware and tools; 


e Understand the “wrap around” needed to 
get positive results using the courseware 
(implementation research); 


e Determine consistency of positive outcomes 
when implemented at scale; and 


e Assess cost efficiency. 


The remainder of this report first addresses the 
development and scaling of grantees’ NGCC 
products and then addresses the latter three 
elements of the evaluation framework. 
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NGCC Courseware 
Development and Refinements 


The following pages provide brief descriptions of 
each NGCC grantee’s courseware as well as links to 
web sites with additional information. 


Acrobatigq - acrobatiq.com acrobatiq 


Launched from Carnegie Mellon University (CMU), Acrobatiq uses an approach to technology-enabled 
learning grounded in the cognitive and learning science research of CMU’s Open Leaming Initiative 
(OLI). Acrobatiq combines an adaptive learning platform, courseware library, and professional services 
to enable higher education institutions to develop, scale, and evaluate personalized learning. 


Courseware offered by Acrobatiq spans business and economics; computing and information 
technology; healthcare and nursing; humanities, social science, and education; and, mathematics and 
science. The courseware can be used in place of a traditional textbook, providing students with a more 
interactive learning experience and educators with real-time data on student learning performance. 
Acrobatiq’s Statistics course, one of several courses being enhanced with NGC grant funding, is the 
most full-featured of the company’s courseware releases to date. 


Acrobatiq’s course design framework aligns course content, activities, and assessments to learning 
outcomes. The courseware includes opportunities for students to demonstrate concept mastery via 
diagnostic, formative, and summative assessments. Practice questions are tailored to each student 
based on the current estimate of that student’s learning state. Students with low levels of mastery are 
given more opportunities to practice, as well as multiple levels of hints and feedback aimed at specific 
mistakes or misconceptions. Interactivity is also a focus of Acrobatiq’s courseware. Their newest 
courseware includes as many as 200 dynamic media elements and interactive exercises, such as 
manipulable images, to enhance learner engagement and understanding. 


As students interact with Acrobatiq’s courseware, learning data are collected, analyzed, and 
reported through an instructor dashboard in real time. Dashboards are organized around 
common questions educators have about student performance and are used to identify skills 
and concepts that students are finding difficult to master, pinpoint at-risk students, and provide 
targeted interventions as needed. 


Acrobatiq’s courseware conforms to learning tools interoperability (LTI) standards to support 
integration with learning management systems (LMS). It is also supported on mobile devices. 
Additionally, Acrobatiq offers an easy-to-use course authoring tools as part of the platform. The 
authoring tools enable educators and instructional design teams to collaborate and customize 
the courseware by incorporating new content using pre-formatted templates and prompts that 
promote sound design and assessment principles. 
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Cerego - cerego.com Cerego 


Cerego provides technology to help users learn more efficiently. The company provides 
customizable courseware for postsecondary education, with offerings in statistics, 
macroeconomics, microeconomics, U. S. history, biology, anatomy and physiology, and sociology. 
In addition, Cerego licenses its technology to publishers, schools, companies and developers to 
help them create personalized learning and training experiences. 


Cerego’s technology draws from the learning science principles of “retrieval practice” and 
“distributed practice” to help students learn and retain foundational concepts. Each Cerego 
module or course unit consists of a “set” of concepts and definitions for which students 
must achieve a target level of retention. The desired retention level can be determined by 
instructors, and the system then prompts students to review concepts at spaced intervals 

to reach and sustain that level. The lengths of these intervals are not fixed, but adapt 
individually to each student depending on their learning and behavior. Students have access 
to a dashboard that displays their activity and progress data. They can access Cerego on 
mobile devices as well as PCs. 


Cerego’s courseware combines its core learning technology with interactive multimedia content 
developed by partners and third parties, such as statistics simulations from the CREATE lab at 
New York University and instructional content from OpenStax and other sources. 


Cerego’s courseware is designed to augment instructor-led instruction rather than as standalone 
course content. Instructors can select and modify modules that fit with their curricula and 
learning objectives. They also have access to dashboards displaying activity and progress data for 
students in their classes. 
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CogBooks: 
CogBooks — cogbooks.com Adaptive Learning 


CogBooks has developed and provided adaptive personalized learning for over 15 years. It offers 
adaptive courseware in a range of scientific and liberal arts topics; major publishers also use 
CogBooks to deliver adaptive courseware to a global audience. 


CogBooks emphasizes the application of cognitive science principles of learning. It aims to 
replicate the actions of an intelligent personal tutor, giving highly personalized support to 
students as they are studying. The core design principle behind CogBooks courses is uses of 
active or effort-based learning with periodic learning checks. Each course is broken down 

into short modules, with learning activities and quizzes that support students’ ability to build 
knowledge and improve retention. At all times, adaptive feedback provides the student with a 
range of recommended activities to choose from, when they need extra help. 


Instructors can use CogBooks courseware in a fully online or blended format, add or change 
content, customize the courseware, override built-in"the grading policies, and generally control 
the content presented to each student. The instructor dashboard allows educators to review the 
performance of individual students as well as the entire class. Instructors can also use a range of 
built-in class management and peer discussion tools. CogBooks integrates via LTI, allowing deep 
linking into LMS course shells and full gradebook integration. 


Building on the success of the program (and with support from the Bill & Melinda Gates 
Foundation) CogBooks has partnered with Arizona State University to build a complete and 
integrated adaptive biology degree program. This program covers 20 biology major courses that 
can be used independently as adaptive courseware or as an integrated whole degree program. 
The adaptive degree delivers a seamless personalized learning journey for each student through 
their whole degree. The Biology Majors Courseware is now available for adoption by any U.S. 
postsecondary institution. 


CogBooks also offers its cloud-based authoring and course-building platform to educators and 
publishers. The entire suite of tools allows authors to build, sell, and deliver adaptive courseware, 
and can be licensed to individual educators, a whole institution, or a publisher. 
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Lumen Learning - lumenlearning.com cs lumen 


Lumen Learning’s courseware combines open educational resources (OER) with adaptive 
personalized learning software. Lumen courseware can be used to teach core content in face- 
to-face (flipped classroom model), blended and online modalities. Courses are available in 
Intro to Business, Macroeconomics, Microeconomics, and Intro to Marketing. Structured in 
modules, the content is presented in the form of text, multimedia, and simulations. Individual 
student learning plans are built as the learning system analyzes data gathered as the student 
works through the content, embedded formative assessments, assignments, and summative 
assessments. The learning plans make recommendations that allow both students and 
instructors to adjust and tailor their activities to support learning. 


Lumen Learning courseware can be integrated into major learning management systems 
including Blackboard, Canvas, and Desire2Learn Brightspace. Faculty members can modify the 
course materials to fit the desired course session length (most often 8, 12, or 16 weeks). 


The courseware also includes tools supporting instructor-student communications. 

Instructors receive automatically generated messages when students are struggling, including 
recommendations about which students would benefit from individualized outreach and where 
they need help. Instructors can use courseware tools to send customizable, automated messages 
with both positive feedback (“great job on that quiz!”) and situation-appropriate notifications 
when a student falls behind. 
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OpenStax -— tutor.openstax.org 
openstax” 


OpenStax Tutor Beta is research-based courseware that integrates with three OpenStax 
textbooks: College Physics, Biology 2e, and Introduction to Sociology 2e. This courseware enables 
instructors to assign reading and homework assignments and includes a comprehensive 
assessment library for each discipline. OpenStax Tutor Beta is designed to enrich the student 
learning experience by integrating interactive elements, such as videos and simulations, and by 
using spaced practice and personalized questions. 


Key features of OpenStax Tutor Beta include performance forecasting and assignment analytics 
designed to help instructors target their teaching and to help students engage in extra practice 
where they need it most. The courseware provides immediate feedback to students working 
through reading assignments and, at the end of each assignment, presents students with spaced 
practice and personalized questions. 


Instructors can use OpenStax Tutor Beta in online courses or in blended formats. OpenStax Tutor 
Beta integrates with Blackboard, Canvas, Desire2Learn, Moodle, and Sakai learning management 
systems (LMSs). 


A nonprofit based at Rice University, OpenStax provides free, peer-reviewed, openly licensed 
textbooks and low-cost courseware to help students learn. 
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SMART 
Smart Sparrow - smartsparrow.com SPARROW 


Smart Sparrow is a learning technology company inspiring the next wave in digital learning and 
assessment. The Inspark Teaching Network is led by a partnership between Smart Sparrow and 
Arizona State University's Center for Education Through eXploration, which was initiated in 
response to the Next Generation Courseware Challenge of the Bill & Melinda Gates Foundation. 


Inspark represents a collaboration between institutions of tertiary and secondary education, 
scientists, researchers, and education experts all working to transform teaching and learning. 
The network has created a series of courses enabling educators to collaborate, build, and share 
next-generation learning experiences. The courseware harnesses personalized, interactive 
digital learning technologies and focuses on active learning designed to spark student curiosity, 
discovery, and exploration. 


Organizing learning as an exploration of the unknown is a hallmark of the Inspark Smart Course 
curricular design. Whole courses are organized around big driving questions, like whether we 
are alone in the universe. Students can access the Smart Sparrow platform directly on the web 
or through an institution’s Learning Management System (LMS). Inspark teaching partners are 
now reaching almost 10k students a semester from some 60 institutions, at both 2-and 4-year 
colleges, with significant growth. 


Courses created by Inspark and partners on the Smart Sparrow platform stand out in the market 
by their extensive use of virtual simulations, promotion of learning-by-doing, and high-quality 
graphics for science education. For example, HabWorlds features the Build-a-Planet simulation 
where students manipulate the level of properties such as gas, ice, and rock to build a planet 
that can sustain life. BioBeyond includes an immersive fly-through into the cell, which can also 
be offered in virtual reality. The suite of web-based software tools includes an adaptive learning 
platform and tools for developing learning content and applications, deploying that material 

to students, and analyzing how students learn from their responses to the material. Adaptive 
feedback is provided in different forms such as a hint, a video, a graph, or additional material 
triggered by a student’s response to a question, the time spent on a screen, or the number of 
question attempts. 


Instructors can use Inspark courseware to teach an entire course or they can select individual 
modules and activities for their classes. Because it’s built on the Smart Sparrow platform, 
instructors have a dashboard which allows educators to review the performance of individual 
students or the entire class. Smart Sparrow’s suite of tools can be licensed at the level of a single 
class, multiple classes, department, or institution. 
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Open 
Open Learning Initiative (OLI) at Stanford - S) anita 


oli.stanford.edu 


The Open Learning Initiative (OLI) at Stanford is a grant-funded organization housed at Stanford 
University that offers openly-licensed, adaptive online courseware developed by and for 
institutions of higher education. OLI’s stated mission is to provide high-quality online courses 
that are based on human learning research and that function as laboratories for continued 
learning sciences research. The open and free versions of the OLI at Stanford courses in 
statistics and computer science are available on Stanford’s Open edX platform. OLI at Stanford 
collaborated with OLI at Carnegie Mellon on course design, improvement, and delivery for all of 
their partners. The academic versions of the courseware are available at oli.cmu.edu. 


For the NGCC program, OLI implemented three statistics courses: Probability and Statistics, 
Statistical Reasoning, and Concepts in Statistics. All three statistics courses include expository 
text, simulations, case studies, comprehension tests, and the StatTutor lab. Complete 
instructions are included for the statistical software packages Microsoft Excel, Minitab, R (open 
source), TI calculator, and StatCrunch. Interactive learning exercises provide opportunities for 
practice and self-assessment and give students feedback while they are working on activities 
(and before being assessed for a grade). 


A distinctive feature of Stanford’s OLI statistics courseware in the incorporation of mindset 
interventions designed to help struggling learners develop a “growth mindset” that encourages 
them to develop their expertise through hard work on challenging tasks. Content and 
assessments are tied to learning objectives and skills. These skills feed a newly developed Open 
Analytics Research Service (OARS) that provides actionable feedback to learners, instructors, 
course authors and learning researchers using openly available and testable predictive models. 
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The NGCC Courseware 
Pilot Phase 


The NGCC grants were structured to foster a 
data-driven, continuous improvement approach to 
courseware design and implementation. Recognizing 
that early versions of a product almost always have 
some “kinks” and that faculty may need some 
experience to learn how to make courseware an 
effective part of their classes, the foundation set up 
the first half of the three-year grants as a time for 
development and pilot testing. Grantees were expected 
to have at least early versions of their courseware 
products in use in college classrooms in academic 
year 2015-16 and to collect data from those early 
implementations that could be used to guide product 
refinements and to develop implementation guidelines 
for further work. Further, grantees were required to 
document the findings from their pilot studies and the 
ways in which they would use those findings in their 
subsequent activities. 


All seven of the grantees met the milestone of having 
versions of their courseware in the field by spring 2016. 
Although they had limited time to make software changes 
before the next round of courseware implementations 

in academic year 2016-17, the NGCC grantees made 
significant changes in their products and/or their 
implementation guidelines in response to things they 
learned from pilot testing. One grantee, for example, 
changed the way their system handles adaptive practice. 
They moved from using a learning curve analysis of 
student performance to employing dynamic skill graphs 
developed through machine learning techniques. They 
also changed the way that their product introduces the 
concept of adaptive practice to students after finding a 
low rate of completion for the recommended practice 
items among pilot students. 
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Another grantee, after finding that students using their 


product on mobile devices made more progress than 
students not using mobile devices, decided to make 
their product a stand-alone mobile application. To 
handle the problem of synchronizing activity within their 
product with the pace of classroom instruction, this 
grantee created an option for the instructor to lower the 
mastery level for their learning modules and reduce the 
number of items in each module so that all students 
would be more likely to master a module within a week. 
They also deleted quantitative problems from their 
instructional modules. To further motivate students, 
they developed a student view of the learning goal, 
their current level, and the progress they made after 
every review session. Finally, they developed tools that 
instructors could use to edit the digital content to fit the 
particulars of their course. 


A third grantee responded to students’ desire for more 
guidance on how to use the courseware by developing 
user guides and additional supports including the 
ability for students to communicate with their instructor 
through the courseware. They also increased the 
number of assessment items and added more variety 
in item formats. To provide more support to instructors 
using their digital content in a blended course, they 
developed a set of course progress reports that 
instructors can download and use to inform their 
choice of what to work on during class time. 
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One grantee undertook a number of product 
refinements after their first implementations in the 

fall of 2015. Most importantly, they improved the 
alignment between their course content and their 
embedded assessments so that their mastery feature 
would function better. They also added a student 
orientation module to provide students with guidance 
on how to use the courseware. They reworked their 
multi-select assessment items as regular multiple- 
choice questions and gave instructors the ability to 
edit quiz items and to allow multiple quiz attempts if 
they wished. Finally, they developed a personalized 
study plan feature to encourage students to use the 
courseware resources to attain mastery. 


In response to the pilot finding that only a minority of 
students used their product’s concept learning modules 
at the end of book sections, one grantee is now 
recommending to instructors that they make weekly 
use of these modules a course requirement. They 

have also added an explanation of the value of spaced 
practice for students and in the future they will make it 
possible for students to repeat an activity. They have 
added links to relevant content on every page of their 
online text so students can see additional content as 

an overlay. Finally, they have revised their assessment 
items by removing those that were too easy or too hard 
and adding an explanation for two-step free response- 
multiple choice items (where students first respond to 

a question by filling in a text box and then choose the 
response option most like their free-response answer for 
a multiple-choice version of the question). 


Another grantee removed the learner success 

module that students responded poorly to from their 
courseware and is in the process of reworking it. They 
have also redesigned their continuous scoring system 
and have developed implicit “triggers” for remediation 
when student performance suggests it is needed. 
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Finally, one grantee implemented the training for 
instructors new to teaching with their courseware 
that one of their pilot sites requested and added 
assessment items to their test bank. They also made 
changes that enable instructors to author and edit 
content. They are exploring the possibility of replacing 
the prompts for students to compare their answer 
with that of an expert with a report to the student of 
the system’s estimate of his or her learning state. 


The NGCC grantees made significant efforts to improve 
their courseware to address the needs and concerns 
that emerged during the first year of implementing their 
early-stage products. In our experience, the kinds of 
challenges they encountered during their first round of 
implementations are quite typical, and other developers 
should expect similar hiccups and the associated 
insights to emerge during their own early field testing. 
These findings underscore the importance of early 

pilot testing in real classrooms. The NGCC grantees 
should be commended for their openness concerning 
the areas for improvement revealed in these early 
implementations as well as for their efforts to make 
rapid improvements in their products even as they 
were working against aggressive scaling targets. 


General themes running through the NGCC pilot 
findings described above highlight areas where 
it is common for early learning technology 
products to have room for improvement: 


¢ More orientation needed to how to use the 
courseware 


e Assessment items that were confusing or had 
inappropriate difficulty levels 


¢ Lack of ability for instructors to modify 
content, assessment items, and mastery 
criteria 


e Perceived disconnects between courseware 
content and course learning objectives 


e Tendency for both students and instructors 
to take shortcuts wherever the system 
allowed them 
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Scaling of NGCC 
Courseware 


All of the NGCC grantees had prior product users 
who could be leveraged as a source of potential 
adoptions for their refined or new products being 
developed with NGCC funding. The grant terms 
challenged grantees to go much further, however, to 
attract enough new adopters to have over 150,000 
cumulative student users by the end of 2017. 
Moreover, the foundation was clear that it wanted 
to see large numbers of low-income and under- 
represented minority students experiencing the NGCC 
products. SRI compiled course section enrollment 
data provided by NGCC grantees on a semiannual 
basis between spring 2015 and fall 2017. Use of 
the NGCC products by postsecondary institutions, 
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instructors, and students over the course of the grant 


term, as tabulated by SRI, is shown in Tables 1-4 and 
Figure 1. It should be noted that some of the NGCC 
products had additional users not included in the 
tables and figure presented here because supporting 
use information (institution, course title, instructor 
name, term) was not sent to SRI. 


Table 1. Number of Institutions Participating in NGCC, by Sector 


janes Public 2-year es by Online 4-year Total 
Acrobatiq fe) 10 14 9 30 
Cerego 8 3 1 _ 42 
CogBooks 2 2 ez Z D) 
Lumen Learning 3 7 E 1 11 
OpenStax 104 137 ac 1 319 
Smart Sparrow a 16 2 - 25 
Stanford 31 39 24 1 95 
TOTAL? 147 192 107 3 


Note: Online designates colleges offering all programs online. 


a Total number of unique institutions; columns will not add to Total in cases where the same institution implemented products from 


multiple grantees. 
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Table 2. Number of Instructors Participating in NGCC, by Sector 


Aer Public 2-year pubes Online 4-year a Key r-)| 
Acrobatiq 22 35 Sie 18 107 
Cerego ts 6 8 - 29 
CogBooks 16 - - - 16 
Lumen Learning 32 44 42 118 
OpenStax 185 190 88 8 416 
Smart Sparrow 20 OS S - 60 
Stanford fal 165 53 6 295 
TOTAL? 303 455 185 62 


Note: Online designates colleges offering all programs online. 


@ Total number of unique institutions; columns will not add to Total in cases where the same institution implemented products from 


multiple grantees. 


As can be seen in Table 3, the total documented 
enrollment in NGCC courses during the timeframe 
of the grants was just under 140,000. To ascertain 
the number of low-income students experiencing 
NGCC courseware, SRI requested that course 
enrollment data be disaggregated by student status 
with respect to eligibility for a Pell grant. This level of 
detail was not available for many course enrollments, 
however, and the numbers of Pell students 

shown in Table 3 in the vast majority of cases are 
estimates derived by applying the percentage of Pell 
students in a college as reported to the Integrated 
Postsecondary Education Data System (IPEDS) 

of the National Center for Education Statistics to 
the course enrollment total submitted by a grantee 
or college. In terms of reaching the intended low- 
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income student audience, an estimated 30% of 
students in course sections using NGCC courseware 
were Pell grant recipients. 


Table 4 shows the NGCC student enrollment data 
disaggregated by minority status. In this case, all 
numbers are estimates based on IPEDS ethnicity 
percentages, and five higher education institutions that 
did not report data to IPEDS are excluded from the 
enrollment totals. Here we see that an estimated 36% 
of students experiencing the NGCC courseware were 
from ethnicity groups that historically have been under- 
represented in higher education. 
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Table 3. Number of Students Documented as Experiencing NGCC Courseware, 
2015-17, by Grantee, Institution Type and Pell Status 


eA UL 0) | omme.co\'{=y-| a pO] 0) ema -¥- 1g Private 4-year Online 4-year 


Grantee 
IN Key a I Key ae 
Pell Pell Pell Pell 
Acrobatiq 851 2,540 361 1,188 1,440 Distt WO, BESra WGA  S4bisi7s) 
Cerego® ial OS OOS BS} 274 393 B25 - - 1,51 2,694 
CogBooks? p24 AW e298 - - - - - - 1,244 1,298 
Lumen 
Learning 1085 1,628 1,751 2,960 - - B56 1,362 4,189 5,648 
OpenStax 4,257 9,926 3,494 9,100 1,401 2,894 4 24 9156 21,944 
Smart 
Sparrow 1,344 2,080 Tes 4], 1x8} Bir aa - - Devel Cro40 
Stanford/OL| 3,016 4,908 3,590 138,914 3,093 7,918 110 Le. 9,809 27,062 
TOTAL 13,402 23,973 9,972 28,619 6,394 13,991 11,733 30,282 41,501 96,865 


Note: Student numbers based on course section- and student-level enrollment data sent to SRI by grantees. 
@ Cerego declined to provide enrollment data for fall 2017. 


» Except for CogBooks, all Pell values are estimates based on applying 2013-14 IPEDS data on percentage of Pell students to course 
enrollment data. CogBooks Pell values are based on institutional data (not estimates). 


Table 4. Estimated Number of Under-represented Minority and Other Students 
Experiencing NGCC Courseware, 2015-17, by Grantee and Institution Type 


Public 4-year Public 2-year Private 4-year 


Grantee 


Acrobatiq 799 2,592 742 807 DIZ 1,836 7,991 29,312 11,6538 34,547 
Cerego? 1607 11,443} 223 104 387 ‘SGiil - - DANCY 2,078 
CogBooks? 1,671 871 - - - - - - 1,671 871 
Lumen 

Learning 1,869 1,040 1,847 1,276 - - 1166 1,030 4,901 3,346 
OpenStax ie low Mae AY hols ato | 0° MES Ol sto) 1,267 3,028 14 14 10,147 20,885 
Smart 

Sparrow 1,729 1,694 878 1,028 72 72 - - 2,679 2,794 
Stanford/OL] 3,597 4,302 8,937 8,472 DV 25) 8,884 209 228) 14,868 21,881 
TOTAL® 15,579 21,700 17,136 19,772 5,972 14,351 9,399 30,579 48,086 86,402 


Note: Student numbers based on course section- and student-level enrollment data sent to SRI by grantees. URM = Under-represented 
minority (African American, Hispanic, Native American, Pacific Islander). 


* Cerego declined to provide enrollment data for fall 2017. 


’ Except for CogBooks, all URM values are estimates based on applying 2013-14 IPEDS data on percentage of URM students to course 
enrollment data. CogBooks Pell values are based on institutional data (not estimates). 


° Totals do not include enrollments from institutions that implemented NGCC courseware but were not found in IPEDS data. 
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Scaling Trends 


The number of students enrolled in courses using 
individual grantees’ products are shown by academic 
term in Appendix A. Across the NGCC portfolio, 
documented student enrollment in course sections 
using the products increased term-over-term from 
spring/summer 2015 (when it was 7,745) through 
spring/summer 2017, when it reached 42,456. 


NGCC grantees approached scaling of their products 
in different ways. Some grantees did much of 

their scaling by approaching faculty members who 
had used one of their products in the past. The 
advantage of leveraging existing relationships in 


SRI Education 


this way was that agreements to adopt the NGCC 
version of a product could be reached quite quickly. 
The disadvantage was that each adoption agreement 
brought only as many students as that particular 
faculty member taught. Other grantees focused on 
securing agreements for institution-wide adoption 
with individuals at higher levels of the college or 
university (e.g., department chairs or provosts). 
These agreements were much more laborious to 
obtain, but had the advantage of bringing along 
multiple instructors and of being less dependent on 
any one faculty member. 


Table 5 shows the ratio of instructors to institutions 
for each grantee and market sector. 


Table 5. Ratio of Instructors to Institutions Participating in NGCC, by Sector 


JLT Public 2-year IES Online 4-year Total 

4-year 4-year 
Acrobatiq 2.44 3.50 3.36 6.50 3.34 
Cerego 1.88 2.00 8.00 - 2.64 
CogBooks 8.00 - - - 8.00 
Lumen Learning 10.67 6529 2 42.00 10.73 
OpenStax 1810) 1.39 1.39 3.00 | 248} 
Smart Sparrow 2.86 2.19 50) - 2.40 
Stanford 2.29 4.23 P| 6.00 Soll 1 
NGCC Average? 2.06 2.37 1.73 20.67 2.24 


@ Each institution’s number of unique instructors per institution is equally weighted in the row Total and NGCC Average. 
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The sector breakdown of data on instructors per 
institution suggest that online institutions are more 
likely than others to adopt courseware on an 
institutional level. The smallest instructor:institution 
ratios were found for 4-year private colleges. Among 
public institutions, two-year colleges appear to be 
doing somewhat broader adoptions than four- 
year colleges are, but the difference is not large. 


Next, we categorized the NGCC grantees’ scaling 
efforts as primarily “retail” (going through individual 
faculty members) or “institutional” adoption 
approaches, based on the ratio of instructors 

per institution and information gleaned from 
communications with grantee staff. This classification 
resulted in a fairly even split of the grantees with 
three classified as pursuing institutional adoptions 

as their primary scaling strategy and four classified 
as using primarily a “retail” scaling approach. We 


SRI Education 


then computed the average cumulative number of 
students using the courseware for each term for each 
of the scaling strategy groups. The results are shown 
in Figure 1. 


Based on the experience of the NGCC grantees, 
efforts to scale courseware to higher education 
institutions can start faster by reaching out directly to 
individual faculty members. The fact that a number of 
the grantee organizations had an existing individual 
customer base for earlier products no doubt made 
this easier. Institutional adoptions require more effort, 
and it was only in the latter portion of the grant 
period that institutional adoption strategies 
started to show a steeper growth rate. It should 
be noted also that there are other advantages to 
promoting courseware at the institutional rather than 
individual faculty member level, as will be discussed 
later in this report. 


Figure 1. Average cumulative number of NGCC courseware users by scaling strategy 
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Issues in Courseware 
Implementation 


Many implementation issues were uncovered 

during the courseware pilot phase and through 
conversations with instructors who were early 
adopters. Others came out during researchers’ visits 
to implementing classrooms using pilot versions of 
the software and from instructor responses on the 
instructor Survey. Information from these various 
sources is synthesized and summarized below. 


e In some cases, instructors and/or students 
wanted more guidance on how to use the 
courseware before starting to work with it. 


Usability issues were not uncommon 

but were generally not severe enough to 
curtail courseware use. The most commonly 
reported issues were problems logging into 
the courseware, incompatibilities with existing 
computer hardware or software, and confusing 
dashboards or system interfaces. 


Many students did not use courseware- 
embedded review functions as intended. 
One grantee found, for example, that their 
system’s recommendations for adaptive practice 
were ignored by nearly 40% of their beta product 
student users. Another had system log data 
showing that practice modules at the end of 
their electronic book chapters were skipped by 
many students. On the instructor survey, 39% 

of NGCC instructors reported students using 
the courseware in an unintended manner, such 
as exploiting features of the system rather than 
learning the material, as an implementation 
barrier. 
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Instructors did not always leverage all of the 
courseware’s capabilities. For example, one 
grantee’s system log data showed that during 
their pilot testing only 7% of the messages their 
system recommended that instructors send to 
students actually got sent. 


Using courseware products on mobile 
devices can limit use of interactive media 
or ability to work on complex problems. 
Plans for embedding some of these features 
in versions for mobile platforms were dropped 
because of usability issues. 


Some instructors want the capability to 
modify the courseware. Alignment between 
courseware content and course objectives 
appeared to concern some instructors. This is a 
potential issue for any higher education learning 
technology that incorporates academic content. 
Instructors also wanted to be able to modify 
courseware mastery criteria and pacing that they 
felt were inappropriate for their students. 
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NGCC Courseware 
Outcomes for Students 


We conducted a series of analyses to estimate the 
potential impacts of the courseware products involved 
in NGCC on student course outcomes. While SRI and 
the grantees collaborated in creating impact research 
plans, ultimately the impact study research designs 
were limited to protocols that institutions and their 
instructors were willing to enact. None of the higher 
education institutions viewed randomly assigning either 
students or instructors to courseware and business-as- 
usual course sections as feasible given their standard 
practices for course enrollment. To the extent possible, 
SRI controlled for differences in student characteristics 
that might affect course outcomes in conducting an 
impact analysis for each combination of grantee, 
institution, course, and academic term of courseware 
use (e.g., use of Grantee A’s courseware in Biology 

101 at Creative College in fall 2016). Each of these 
combinations is referred to as an “implementation 
dataset” in the discussions that follow. 


All the implementation datasets submitted by higher 
education institutions and grantees were screened 
for quality prior to conducting an impact analysis. The 
criteria for including a dataset in the NGCC impact 
analyses were: 


e submission of data for both a treatment 
(courseware-using) and a comparison group; 


© common end-of-course outcome measure; 


© common prior achievement or pretest measure 
for both treatment and comparison students 
(usually student’s cumulative college GPA was 
submitted); 


e at least 30 students in each of the treatment and 
comparison groups; and 
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e baseline equivalence (not more than .25 
standard deviation difference between treatment 


and comparison students on any student 
characteristic or pretest measure). 


SRI requested the following student-level variables 

in the datasets for impact analysis: gender, race/ 
ethnicity, Pell status, age, first-generation college 
goer, enrollment status (full- or part-time), cumulative 
college GPA, prior attempts in this course, and prior 
achievement measure (Such as a math placement 
test). Those student-level variables reported for most 
students were used in the analytic model applied 

to a dataset. Very few datasets included adequate 
data on a prior achievement measure other than 
cumulative college GPA. We encouraged grantees 

to work with the higher education partners to set 

up impact studies using a common end-of-course 
assessment across treatment and comparison 
groups. This actually happened in fewer than five 
cases, however. As external organizations, the NGCC 
grantees did not have the degree of influence needed 
to persuade academic departments to change their 
practices with respect to letting individual faculty 
choose how to assess learning. For this reason, we 
used course grade as the common outcome measure 
for almost all datasets. 
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Initially, we attempted to obtain both initial enrollment 
figures and the final enrollment after the course 

drop deadline for each course section to enable 
calculations of attrition rates so they could be 
compared for treatment and comparison conditions. 
After the first few terms, however, this request was 
dropped because of the very low rate of reporting 
compliance and the burden this request placed on 
higher education institutions. 


In many cases, a course was taught using the NGCC 
product in some course sections and without the 
product in other sections during the same academic 
term (concurrent implementation). In other cases, the 
comparison data for the course taught without the 
NGCC courseware came from prior terms (historical 
data) or from one prior or following academic term 
(lagged comparison). 


In all, NGCC grantees and the higher institutions 
working with them submitted 77 implementation 
datasets to SRI. In addition to resolving any issues 
with data labelling or missing data, SRI analysts 
performed a first review to check for any intractable 
problems with the research design. Of the datasets 
submitted to SRI, 8 had to be eliminated because 

of a serious confound in the study design (e.g., the 
courseware condition was at one college while the 
comparison data came from one or more different 
institutions or one condition had a course term twice 
as long as that of the other condition). Education 
technology companies, especially those in “start-up” 
mode, typically lack the internal research expertise to 
guide higher education institutions in setting up good 
study designs, and as noted earlier, they often lack 
the clout to enforce strong designs even when they 
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are confident about what needs to be done. Another 
82 implementation datasets had to be eliminated 
because of inadequate sample size, and 9 were 
eliminated because of a lack of baseline equivalency. '® 


A kind of confounding running throughout the 
datasets to various degrees was that between 
treatment condition and course section instructors. 
Particularly when the courseware and comparison 
sections were both run during the same academic 
term, instructors did not want to be teaching different 
sections in different ways so the two conditions 
usually had different instructors. Although we could 
not eliminate this confounding, we did document 
the extent to which instructors in the two conditions 
were the same, partially overlapping, or completely 
different so that we could analyze this study design 
feature as a potential moderator of courseware 
impacts (as will be discussed below). 


After screening, we had a total of 28 NGCC impact 
study datasets for analysis and synthesis. Table 6, 
below, shows the demographic characteristics of the 
students in these impact study data sets overall and 
by college type. Because some of the studies used 
multiple prior cohorts to estimate the historical average 
performance for the course, there were more students 
in the comparison condition than in the treatment 
condition that experienced the NGCC courseware. In 
both conditions, more than half of the students whose 
data were used in impact analyses were drawn from 
under-represented minorities and over 35% were Pell 
grant students. Across all the impact studies with Pell 
status data, a somewhat higher proportion of students 
in the treatment condition than in the comparison 
condition were Pell students (45% versus 36%). 


19 The baseline equivalency standard applied in the evaluation was that recommended by the What Works Clearinghouse (students in the 
conditions being compared varying by no more than .25 standard deviation). There was one additional case where the baseline equivalence 
test was failed, but the dataset was large enough that we could use the subset of data for full-time students in the course, which met both the 


sample size and the baseline equivalence criteria. 
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Table 6. Inclusion of Pell and under-represented minority students in NGCC impact 


study datasets, by institution type 


Treatment Sections 


i AVG of 


Comparison Sections 


Pell URM 


Number Percent 


Public 2-year (PO2a 674 44% 831 
Public 4-year 2,947 UUs 51% 1,908 
Private 4-year 457 255) 56% 195 
Online 4-year 3,041 ~712 ~384% 1,596 
TOTAL? 7,966 ~3,156 ~40% 4,530 


Note: URM = Under-represented minority 


Number Percent 


Number Percent Number Percent 


55% 3,446 1,601 46% 1,738 50% 
65% 5,010 2,507 50% 2,902 58% 
43% 499 269 54% 213 43% 
52% 6,108 ~1,109 ~37% 4,009 66% 
57% | 15,063  ~5,486 ~36% 8,862 59% 


2 Does not include data from institutions that could not identify Pell students. 


Because of the differences among NGCC products 
and the institutions that adopted them, we chose to 
produce impact estimates for individual datasets and 
then to use a meta-analytic approach to summarize 
findings across datasets rather than putting all the 
data in a single analysis. Meta-analysis is useful as 
a systematic, quantitative technique for combining 
data across similar studies to obtain a single impact 
estimate with greater statistical precision than any 
one of the studies provides. Because it is based 

on multiple studies with greater diversity among 
participants, a meta-analysis can better support 
extrapolation to the general population. It is also 
useful in exploring factors that influence the size of 
the impact in different implementations. 
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Table 7 shows the mean completion rate for 
treatment and comparison course sections prior to 
adjustment for differences in student characteristics 
for the 28 datasets. We attempted to run impact 
analyses correcting for differences in student 
characteristics for these course completion data 
(dichotomized as 1 = completed and O = did not 
complete). However, because of the small amount of 
variation in this outcome (as most students complete 
the course successfully), the statistical model was a 
poor fit to the data, prompting us to rely on analyses 
of course grades to estimate courseware impacts. 
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Table 7. Unadjusted Completion Rates for Courses in NGCC, by Grantee and IHE Type 


Grantee 


IHE Type 


Course/ Department 


Intro to Psychology/ 


Acrobatiq Online 4-yr Psychology 
Macroeconomics/ 
Cerego Private 4-yr = COnGrniCS 
: The Living World/ 
CogBooks Public 4-yr Biology 
es General Biology/ 
CogBooks Public 4-yr Biology 
ete General Biology/ 
CogBooks Public 4-yr Biology 
CogBooks Average” 
Principles of 
Lumen Online 4-yr Marketing/ Business 
; Intro to Business/ 
Lumen Online 4-yr Business 
Principles of 
Lumen Online 4-yr = Macroeconomics/ 
Economics 
Principles of 
Lumen Online 4-yr Microeconomics/ 
Economics 
Intro to Business/ 
Lumen Public 2-yr Bieinieee 
Macroeconomic 
Lumen Public 2-yr Principles/ 
Economics 
Intro to Business/ 
Lumen Public 4-yr EVemese 
Intro to 
Lumen Public 4-yr Microeconomics/ 
Economics 
; Principles of 
Lumen Public 4-yr_— Marketing/Business 
Intro to Business/ 
Lumen Public 2-yr Baemecc 
Principles of 
Lumen Public 2-yr Macroeconomics/ 
Economics 


Course 
Type 


Gateway 
Gen Ed. 
Gen Ed. 


Gen Ed. 


Gen Ed. 


Gateway 


Gen Ed. 


Gen Ed. 


Gateway 


Gateway 


Gen Ed. 


Gen Ed. 


Gateway 


Gateway 


Gateway 


Gen Ed. 


Modality 
Contrast 


Online vs 
online 


Online vs 
online 


Blended/ 
hybrid vs F2F 


Blended/ 
hybrid vs F2F 


Blended/ 
hybrid vs F2F 


Online vs 
online 


Online vs 
online 


Online vs 
online 


Online vs 
online 


Online vs 
online 


Blended/ 
hybrid vs F2F 


Blended/ 
hybrid vs F2F 


Blended/ 
hybrid vs F2F 


Blended/ 
hybrid vs F2F 


Blended/ 
hybrid vs F2F 


Blended/ 
hybrid vs F2F 


Treatment 
Completion 
Rate? 


84 


18 


wall 


se) 


93 


82 


ail 


.69 


.60 


Comparison 
Completion 
Rate? 


52 


90 


all Ut 


aol 


39 


51 


83 


84 


86 


87 


64 


61 


90 


fs 


aol 


.60 


a2 


a Completion rates do not control for other predictors of student success, such as prior academic achievement. Completion rates should 
therefore be interpreted cautiously. We did not perform statistical tests for uncontrolled completion rates due to interpretation concerns. See 


Table 8 for impact estimates that include statistical controls for prior achievement and other predictors of student success. 


b Grantee averages are weighted such that each course at each institution counts equally regardless of the number of students or sections. 
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Table 7. Unadjusted Completion Rates for Courses in NGCC, by Grantee and JHE Type, (cont'd.) 


Grantee 


IHE Type 


Course/ Department 


Course 
Type 


Modality 
Contrast 


Lumen Public 2-yr 
Lumen Online 4-yr 
Lumen Online 4-yr 
Lumen Online 4-yr 
Lumen Online 4-yr 
Lumen Average® 

Smart 
Sparrow Public 4-yr 
Smart ; 
Sparrow Public 2-yr 
Smart 
Sparrow Public 4-yr 
Smart 
Sparrow Public 2-yr 
Smart 
Sparrow Private 4-yr 
Smart Sparrow Average? 
Stanford/ 

OL| Public 2-yr 
Stanford/ 

OL| Online 4-yr 


Stanford/ OLI Average? 


NGC Overall Average 


Principles of 
Microeconomics/ 
Economics 


Marketing Principles/ 
Business 


Intro to Business 
and Management/ 
Business 


Principles of 
Macroeconomics/ 
Economics 


Principles of 
Microeconomics/ 
Economics 


The Living World/ 
Biology 


Principles of 
Biology/ Biology 


General Education 
Biology/ Biology 


Biology Concepts/ 
Biology 


Survey of 
Bioscience/ Biology 


Intro Statistics/ 
Mathematics & 
Statistics 


Intro to Statistics/ 
Mathematics & 
Statistics 


Gateway 


Gateway 


Gateway 


Gen Ed. 


Gateway 


Gen Ed. 
Gen Ed. 
Gen Ed. 
Gen Ed. 


Gen Ed. 


Gateway 


Gateway 


Blended/ 
hybrid vs F2F 


Online vs 
online 


Online vs 
online 


Online vs 
online 


Online vs 
online 


Blended/ 
hybrid vs F2F 


Blended/ 
hybrid vs F2F 


Blended/ 
hybrid vs F2F 


Blended/ 
hybrid vs F2F 


Blended/ 
hybrid vs F2F 


Blended/ 
hybrid vs F2F 


Online vs 
online 


Treatment Comparison 
Completion Ofeynaye)(=14(0)4) 
Rate? Rate? 
.66 Me 
74 olf if 
XO) 61 
.86 20 
18 .86 
WAL 73 
83 83 
.60 off | 
.80 14 
19 oS 
85 88 
AMS .78 
84 ole 
alll .69 
81 ofl 
75 14 


a Completion rates do not control for other predictors of student success, such as prior academic achievement. Completion rates should 
therefore be interpreted cautiously. We did not perform statistical tests for uncontrolled completion rates due to interpretation concerns. See 
Table 8 for impact estimates that include statistical controls for prior achievement and other predictors of student success. 


b Grantee averages are weighted such that each course at each institution counts equally regardless of the number of students or sections. 
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Figure 2 shows the effect size for each of the course. In 4 implementations, students in the business- 
implementation datasets in our analysis, and Table as-usual version of the course earned higher grades. 
8 provides the statistical details for the 28 impact In the other half of the datasets, student grades in the 


estimates. In 10 of the 28 datasets, students using the two conditions were equivalent after controlling for 
NGCC courseware earned significantly higher grades differences in student characteristics. 
than those in the business-as-usual versions of the 


Interpreting Effect Sizes 


For each courseware impact study, analysts first used a statistical model (ordinary least squares 
regression) to control for any differences between courseware-using students and those in the 
comparison course sections in terms of prior achievement and other characteristics potentially 
related to course outcomes (such as ethnicity or enrollment as a full-time or part-time student). 
Using the corrected course grades generated by the statistical model, analysts then computed 
an effect size. Effect sizes in these analyses are the difference between the treatment and the 
comparison group model-adjusted means for course grade expressed in standard deviation 
units. (See Appendix C for more information about the statistical modelling.) An effect size with 
a+ sign means that grades were higher for the courseware-using students; an effect size with a 
-sign means that students in comparison course sections had higher grades. 


When testing the hypothesis that the two course section types differed in terms of course 
grades, it is important to keep in mind that we are working with a sample that represents a 
larger population of interest (here, the population of all students at a particular college taking 
the designated course implemented with courseware versus the population taking the same 
course at the same institution without courseware). To control for the risk of concluding that 
there is a difference between coursewareusing students and other students when there really 
is not, analysts set a criterion or level for the acceptable rick of concluding there is a difference 
based on outcomes for the samples in the study when there really is not one for the populations 
from which the samples were drawn. The graphs that follow use the conventional standard for 
statistical significance of having less than a 1 in 20 chance of concluding that two means are 
different when the underlying populations are equivalent (p < .05). Some tables later on in the 
report also indicate which impact estimates meet more stringent standards such as only a1in 
100 chance of a false positive conclusion (p < .01) or a 1 in 1,000 chance (p < .001). 


Statistically significant impact estimates are shown in boldface in the graphs that follow. For 
those impact estimates that are not shown in boldface, we cannot be confident that the observed 
difference in course grades is not attributable to chance. Note that the lack of a statistically 
significant difference does not mean that we can be confident that there is no difference. Additional studies 
are needed to address the uncertainty in these cases. 
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Figure 2. NGCC Courseware Impact Study Effect Estimates 


Grantee/Institution 
Acrobatiq/WGU 
Cerego/Columbia 
CogBooks/ASU 
CogBooks/SouthernU and A&M 
CogBooks/SouthernU and A&M 
Lumen/Broward 

Lumen/Broward 

Lumen/Broward 
Lumen/Broward 
Lumen/Glendale 
Lumen/Glendale 
Lumen/Pittsburg State 
Lumen/Pittsburg State 
Lumen/Pittsburg State 
Lumen/Tidewater 
Lumen/Tidewater 
Lumen/Tidewater 

Lumen/UMUC 

Lumen/UMUC 

Lumen/UMUC 

Lumen/UMUC 

Smart Sparrow/ASU 

Smart Sparrow/Central Piedmont 
Smart Sparrow/Miami-Dade 
Smart Sparrow/Mohave 

Smart Sparrow/National Louis U 
Stanford-OLI/Canyons 
Stanford-OLI/UMUC 


Total 


-1.00 


_ Favors 
Business-as-Usual 


Course 

Introduction to Psychology 
Macroeconomics 

The Living World 

General Biology 

General Biology 

Principles of Marketing 
Introduction to Business 
Principles of Macroeconomics 
Principles of Microeconomics 


Introduction to Business 
Macroeconomics Principles 
Introduction to Business 
Introduction to Microeconomics 
Principles of Marketing 
Introduction to Business 
Principles of Macroeconomics 
Principles of Microeconomics 
Marketing Principles 

Introduction to Business & Mgmt. 
Principles of Macroeconomics 
Principles of Microeconomics 
The Living World 

Principles of Biology 

General Education Biology 
Biology Concepts 

Survey of Bioscience 
Introduction to Statistics 
Introduction to Statistics 


1.00 


Favors 
Courseware 


The square for each courseware implementation represents the estimated impact (effect size). The size of the square represents the weighting of 
the study in the meta-analysis. The length of the horizontal line through the square represents the confidence interval around the impact estimate. 
The longer the line, the more uncertainty there is around the true impact of the courseware implementation. The diamond at the bottom of the graph 
represents the average effect for the 28 courseware implementations. Its width represents the 95% confidence interval for the average impact. 


Squares to the right of the boldface vertical line represent studies in which students in the courseware sections outperformed students in 

the business-as-usual sections. Squares to the left of the boldface vertical line represent studies in which students in the business-as-usual 
courseware sections outperformed students in the courseware sections. Only those courseware implementations with boldface labels are 
statistically significant (i.e., we can be confident that the difference in average grades for students in courseware and business-as-usual course 
sections did not occur by chance). In courseware implementations that do not have boldface labels, we cannot rule out the possibility that the 
observed difference in courseware and business-as-usual section average grades was attributable to chance. 
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Table 8. Impact Estimates for 28 NGCC Impact Studies 


Treatment Sample Effect Standard Upper 


SUL S2 oe for Analysis? ST) 42 (2) Error Limit ELI 
Acrobatiq ea 1,071 40.221 0.049 40.124 40.318 4.467 
Cando eee ee 396 0.161* 0.070 -0.298 -0.025 -2.315 
CogBooks ASU The Living World 1,259 +0.306"* 0.044 +0.220 40.392 6.977 
CogBooks eee cee one 448 40.306" 0.057 +0.194 0.418+ 5.345 
CogBooks oes ae 522 +0.300 0.074 40.155 +0.444 4.073 
CogBooks Aggregate Impact* 2,229 +0.305*** 0.031 +0.243 +0.367 9.686 
Lumen eee le 109 0.023 0.118 -0.255 +0208 -0.198 
Lumen ee to O44 0.015 0.084 -0.180 +0.149 -0.184 
i Broward Principles of 

Umenean ee wees 257 0.095 0.070 -0.233 +0.043 -1.354 
Lumen ee ae of 365 0.126 0.058 -0.239 -0.012 -2.174 
Lumen a lnironte 90 40.109 0.123 -0.132 +0350 0.884 

Glendale 
Lumen Macroeconomic 79 +0.105 O,125 -0.139 +0.350 0.843 
Principles 
Lumen ee lative 55 40.050 0.158 -0.259 +0359 0.316 
Lumen Sue see 146 #0.235* 0.112 +0.016 +0.454 2.105 
Pittsburg State 2 
Lumen Baas of Marketing 58 40.284 0.152 -0.013 +0.581 1.872 
Lumen Hees Intro to 271 40.189 0.075 -0.007 +0.285 1.860 
Lumen nea es 70 0.358 0.126 -0.606 -0.110  -2.832 
Lumen UP ate ahs 75 0.181 0.126 -0.427 +0.065 -1.441 
Lumen MINING (eiditel etek 190 -0.148 0.084  -0318 +0016 -1.764 


Marketing 


a Some enrolled students could not be included in the analysis because their records were missing key data elements. 
b Bold numbers in column 4 indicate statistically significant differences. 

c Datasets are weighted equally in computing product average values. 

*p =< 0.05; **p =< 0.01; **p = 0.001 
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Table 8. Impact Estimates for 28 NGCC Impact Studies, (cont'd.) 


Treatment Sample Effect Standard Lower 


Grantee IHE/Course 


for Analysis? Size (g)?° Error Limit 


UMUC Intro to 
Lumen Business and 291 0.300*** 0.078 +0.148 +0.452 Rola 
Management 


UMUC Principles of 


Lumen Mabrocconcnics 241 +0.014 0.076 -0.135 +0.162 0.178 

Wi ieee 182 0.247* 0.087 -0.418 -0.076 -2.828 

Lumen Aggregate Impact* 2,720 -0.002 0.046 -0.092 0.087 -0.049 

Smart a +0.325*** 

Sparrow ASU The Living World 150 . 0.084 +0.160 +0.489 3.871 

Se Bie oe ich Calmont 334 0.115 0.076 -0.263 +0.034 -1.509 

Sparrow Principles of Biology 

Su Mia Dade Ceres 309 +0.346* 0.149 +0.055 +0638 2.327 

Sparrow Education Biology 

am a ee 104 40.181 0.106 -0.028 +0390 1.699 

eran National Louis 

Se attow University Survey of 61 +0.172 0.176 -0.174 +0.517 0.975 
: Bioscience 

Smart Sparrow Aggregate Impact* 958 +0.172 0.104 -0.032 +0.375 1.652 

Stanford/ College of Canyons ae 

OL! lie vata CARH SS 498 +0.291 0.069 +0.156 +0.426 4.235 

Be enna i9 94 40.299 0.111 +0.082 +0516 2.704 

Stanford/ OLI Aggregate Impactc 592 +0.293*** 0,058 +0.179 +0.408 5.025 


a Some enrolled students could not be included in the analysis because their records were missing key data elements. 
b Bold numbers in column 4 indicate statistically significant differences. 

c Datasets are weighted equally in computing product average values. 

*p =< 0.05; **p =< 0.01; **p = 0.001 


The average effect size of +.09 means that on that (on average, controlling for what we can control 
average, the grades earned by students in for) a 50th-percentile student in the course taught 
course sections using NGCC courseware were as usual would be predicted to improve to a 54th- 
slightly (and statistically significantly) better percentile student through the use of the NGCC 
than the grades earned by students in sections courseware. This average improvement is very 
without the software. An effect size of 0.09 is modest, but it is clear from Figure 2 that there was 
equivalent to an improvement index of 4. This means great variability across the datasets, sometimes even 
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for the same product and institution (see, for example, 
UMUC results with Lumen Learning). Such variability 
points to the need to look more deeply into the 
conditions and practices associated with more and 
less favorable outcomes for courseware use. 


This variability in impacts is consistent with the 
interpretation that efficacy is not a quality that resides 
in a piece of courseware per se. Rather, efficacy 
results from a particular set of instructional 
practices as they play out in a particular context 
and with particular kinds of students (Means, 
Murphy, & Shear, 2017). Impact estimates will also 
be influenced by the nature of the instruction in 

the business-as-usual course sections to which 
courseware-using sections are compared. The 
variability in impacts for these 28 courseware 
implementations provide a resource for investigating 
the conditions and practices associated with more 
and less positive student outcomes. 
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What student characteristics are 
associated with better courseware 
outcomes? 


Next, we conducted exploratory analyses to examine 
the relationship between NGCC courseware outcomes 
and a set of student characteristics that have been 
associated with differential course performance in 
prior research. Table 9 below shows the number of 
impact study datasets that included sufficient numbers 
of students with data on a characteristic to enable 

us to examine the influence of that characteristic in 
our statistical model, and the proportion of datasets 
including that characteristic for which it was associated 
with differences in treatment group learning outcomes. 


The analysis of relationships between student 
characteristics and NGCC courseware impacts in the 
28 datasets found that: 


e Prior GPA is a strong and consistent predictor of 
grade in courseware-using sections. 


Table 9. Student Characteristics Associated with Courseware Outcomes 


Percent of Studies Where 
Significantly Associated with 
Treatment Outcomes 


Number of Datasets with the 


Student Characteristics Data Elomont 


Cumulative college GPA 96% 26 
Attempted course previously 538% 1S 
Under-represented minority 42% 26 
Pell grant 23% 22 
Age 12% 26 
Gender 11% 28 
Enrollment status (full- v. part-time) 8% 24 
First-generation college goer 8% 13 
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e Students who had attempted the course 
previously and failed to earn a credit had lower 
grades in the majority of the impact analyses 
with this variable in the dataset. 


Under-represented students experienced different 
courseware impacts compared to other students 
in around two-fifths of the studies, meaning that 
in the majority of analyses, courseware impacts 
were equivalent for different ethnic groups. 


There was no difference between impacts for Pell 
and for non-Pell students in three-quarters of the 
studies. Pell students had different outcomes 
than non-Pell students did in the courseware- 
using condition in just under a quarter of the 
studies, but the direction of the difference was 
for Pell students to have better grades as often 
as it was for them to have lower grades. 


Relatively few studies (less than 1 out of 8) found 
associations between age, gender, enrollment 
status, or first-generation college status and 
courseware outcomes. 


In general, once prior achievement (as measured here 
by prior GPA) is controlled for, courseware outcomes 
appear to be similar for students of different ages, 
genders, and status as full- or part-time students. 
The only student characteristics that appear to 
influence courseware outcomes after controlling for 
prior achievement are ethnicity and a prior attempt in 
the course with a D/F/W outcome. 


Under what circumstances are 
NGCC products effective? 


To gain insight into the circumstances that promote 
effective use of courseware and stronger evidence 
of efficacy, we coded each impact study for the 
conditions under which the courseware was used. 
Coded features included: 
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e Type of IHE (2-year versus 4-year college) 
e Course subject area (e.g., statistics, economics) 


e Treatment-comparison modality contrast (i.e., 
blended/online v. face to face; fully online with 
courseware Vv. fully online without courseware) 


e Class size 
e Instructor volunteer status 


e Type of instructor (faculty versus instructor or 
adjunct) 


e Instructor employment status (full- versus part-time) 


We then tested each of these conditions to determine 
whether it was associated with the magnitude of the 
impact of using courseware. Five of these variables 
(IHE type, course subject area, modality contrast, 
class size, and instructor employment status) 

were statistically significant moderators of NGCC 
courseware impacts on student grades. 


Figure 3 shows effect sizes grouped by whether the 
implementation was in a 2-year or a 4-year college. 
The average courseware impact estimate was 
significantly positive for implementations in 
4-year colleges and statistically insignificant 

in 2-year colleges. This finding suggests the 

need to better understand differences in instructor 
practices around courseware use in these two 

kinds of institutions, especially given the importance 
of community colleges as access points for 
postsecondary education for low-income and under- 
represented minority students. However, it should be 
noted that there were only a dozen impact studies 
conducted in community colleges and that most 

of them involved the product of a single NGCC 
grantee. Finding community college partners with 
the organizational capacity to implement adaptive 
courseware and mount credible studies of of its 
impacts was a challenge for the NGCC grantees. 
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Figure 3. Impact Study Effect Estimates by Institution Type 


Institution Name 


Course 


a vene Miami-Dade General Education Biology 
College of Canyons Introduction to Statistics 
Mohave Biology Concepts 
Tidewater Introduction to Business 
Glendale Introduction to Business 
Glendale Macroeconomics Principles 
Broward Introduction to Business 
Broward Principles of Marketing 
Broward Principles of Macroeconomics 
Central Piedmont Principles of Biology 
Broward Principles of Microeconomics 
Tidewater Principles of Microeconomics 
Tidewater Principles of Macroeconomics 
Total 
eet ASU The Living World 
ASU The Living World 


Southern U and A&M 
Southern U and A&M 


General Biology 
General Biology 


UMUC Introduction to Business & Management 
UMUC Introduction to Statistics 
Pittsburg State Principles of Marketing 
Pittsburg State Introduction to Microeconomics 
WGU Introduction to Psychology 


National Louis U 
Pittsburg State 


Survey of Bioscience 
Introduction to Business 


UMUC Principles of Macroeconomics 
UMUC Marketing Principles 
Columbia Macroeconomics 

UMUC Principles of Microeconomics 

Total 
-1.00 1.00 
_ Favors Favors 
Business-as-Usual Courseware 


Boldface indicates that the impact estimate is statistically significant (i.e., we can be confident that the difference in average grades for students in 
courseware and business-as-usual course sections did not occur by chance). In courseware implementations that are not in boldface, we cannot 
rule out the possibility that the observed difference in courseware and business-as-usual section average grades was attributable to chance. 
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Figure 4. Impact Study Effect Estimates by Course Subject Area 


Institution Name Course 
Biology 
Miami-Dade General Education Biology 
ASU The Living World 
ASU The Living World 
Southern U and A&M General Biology 
Southern U and A&M General Biology 
Mohave Biology Concepts 
National Louis U Survey of Bioscience 
Central Piedmont + Principles of Biology 
Total <> 
Business 
UMUC i Introduction to Business & Mgmt. 
Pittsburg State Principles of Marketing 
Tidewater a Introduction to Business 
Glendale Introduction to Business 
Pittsburg State al iia Introduction to Business 
Broward Introduction to Business 
Broward a Principles of Marketing 
UMUC a_i Marketing Principles 
Total > 
Economics 
Pittsburg State —_i— Introduction to Microeconomics 
Glendale —i— Macroeconomics Principles 
UMUC —- Principles of Macroeconomics 
Broward —sH Principles of Macroeconomics 
Broward + Principles of Microeconomics 
Columbia 1 Macroeconomics 
Tidewater —i—- Principles of Microeconomics 
UMUC —— Principles of Microeconomics 
Tidewater === Principles of Macroeconomics 
Total al 
Mathematics & Statistics 
College of Canyons st Introduction to Statistics 
UMUC Introduction to Statistics 
Total => 
Psychology F 
WGU se Introduction to Psychology 
Total a> 
-1.00 -0.50 0.00 0.50 1.00 
Favors Favors 
Business-as-Usual Courseware 


Boldface indicates that the impact estimate is statistically significant (i.e., we can be confident that the difference in average grades for students in 
courseware and business-as-usual course sections did not occur by chance). In courseware implementations that are not in boldface, we cannot 
rule out the possibility that the observed difference in courseware and business-as-usual section average grades was attributable to chance. 
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In terms of course subject area, the 28 impact 
studies dealt with five different academic disciplines: 
biology, business administration, economics, 
mathematics/statistics, and psychology. Figure 4 
shows the impact estimates organized by course 
subject. On average, the implementations of 
courseware in biology, psychology, and math/ 
statistics classes led to higher student course 
grades than those earned in business-as-usual 
classes. In contrast, on average the courseware 
impact appeared to be insignificant for business 
administration and economics classes. This finding is 
consistent with a prior meta-analysis finding stronger 
benefits for blended learning in STEM (science, 
technology, engineering, and mathematics) classes 
than in other subject areas (Vo, Zhu, & Diep, 2017). 
However, it should be remembered that there were 
just a few products or even only a single courseware 
product for each subject area. 


The NGCC courseware implementations varied in 
terms of whether the courseware was incorporated 
into a course taught primarily in a face-to-face 
mode, using some blend of face-to-face and 

online elements, or was entirely online. In addition, 
the comparison course sections to which the 
courseware-using sections were compared varied in 
terms of whether they were taught online or face- 
to-face. Among the impact studies, 12 contrasted 
two online versions of the course (one with the 
NGCC courseware and one without), and the others 
used blended or in one case, online instruction in 
the treatment condition, and instruction that was 
primarily face-to-face in the comparison condition. 
Figure 5 illustrates the variation in estimated 
courseware impacts for these different modality 
comparisons. 


Next Generation Courseware Challenge Evaluation 


SRI Education 


Only those impact studies in which the course 
modality varied between treatment and 
comparison groups (i.e., blended/online vs. 
face-to-face) produced a significantly positive 
impact on average. This is congruent with findings 
reported in a larger meta-analysis performed for 

the U.S. Department of Education (Means et al., 
2010) and reported in the earlier study of adaptive 
courseware by Yarnall, Means, and Wetzel (2016). 

It appears that the best prospects for improving 
student outcomes are associated with moving from a 
traditional face-to-face format to a blended learning 
approach promoting more active learning on the part 
of students. In the NGCC studies it was not possible 
to disentangle the impact of the NGCC courseware 
per se from that of the shift in instructional modality. 
From the standpoint of educators, however, 
identifying a strategy for improving outcomes 
(combining courseware with face-to-face instruction) 
is important even if we cannot quantify the extent 

to which the courseware itself is responsible for 
observed improvements. 


The courseware impact estimate varied also by 
course student:teacher ratio. Implementations 
where the instructors taught groups larger than 
50 students appeared to have more positive 
outcomes than studies where the class sections 
were smaller. Figure 6 illustrates the relationship 
between student:teacher ratio and courseware 
impact. It may be that courseware is particularly 
useful in cases where instructors are dealing with 
more students than they can attend to individually. 

If one assumes that these larger courses were 
lecture-dominated in the past, the move to blended 
learning incorporating adaptive courseware would 
likely increase the amount of active learning, which is 
known to improve learning outcomes. 
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Figure 5. Impact Study Effect Estimates by Modality Contrast 


Institution Name 
Online or Blended/hybrid vs F2F 
Miami-Dade 

ASU 

ASU 

Southern U and A&M 
Southern U and A&M 
College of Canyons 
Pittsburg State 
Pittsburg State 
Mohave 

National Louis U 
Tidewater 

Glendale 

Pittsburg State 
Central Piedmont 


Course 


General Education Biology 
The Living World 

The Living World 

General Biology 

General Biology 
Introduction to Statistics 
Principles of Marketing 
Introduction to Microeconomics 
Biology Concepts 

Survey of Bioscience 
Introduction to Business 
Macroeconomics Principles 
Introduction to Business 
Principles of Biology 


Tidewater Principles of Microeconomics 
Tidewater Principles of Macroeconomics 
Total 
Online vs Online 
UMUC Introduction to Business & Mgmt. 
UMUC Introduction to Statistics 
WGU Introduction to Psychology 
Glendale Introduction to Business 
UMUC Principles of Macroeconomics 
Broward Introduction to Business 
Broward Principles of Marketing 
Broward Principles of Macroeconomics 
Broward Principles of Microeconomics 
UMUC Marketing Principles 
Columbia Macroeconomics 
UMUC Principles of Microeconomics 
Total 
-1.00 1.00 
Favors Favors 
Business-as-Usual Courseware 


Boldface indicates that the impact estimate is statistically significant (i.e., we can be confident that the difference in average grades for students in 
courseware and business-as-usual course sections did not occur by chance). In courseware implementations that are not in boldface, we cannot 
rule out the possibility that the observed difference in courseware and business-as-usual section average grades was attributable to chance. 
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Figure 6. Impact Study Effect Estimates by Class Size 


Institution Name 
Biology 

Miami-Dade 

ASU 

ASU 

Southern U and A&M 

Southern U and A&M 

Mohave 

National Louis U 

Central Piedmont 

Total 


Business 
UMUC 


Pittsburg State 
Tidewater 
Glendale 
Pittsburg State 
Broward 
Broward 
UMUC 


Total 
Economics 


Pittsburg State 
Glendale 
UMUC 
Broward 
Broward 
Columbia 
Tidewater 
UMUC 
Tidewater 
Total 
Mathematics & Statistics 
College of Canyons 
UMUC 
Total 


WGU 
Total 


Psychology 


-1.00 


_ Favors 
Business-as-Usual 
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Course 


General Education Biology 
The Living World 

The Living World 

General Biology 

General Biology 

Biology Concepts 

Survey of Bioscience 
Principles of Biology 


Introduction to Business & Mgmt. 
Principles of Marketing 

Introduction to Business 
Introduction to Business 
Introduction to Business 
Introduction to Business 

Principles of Marketing 

Marketing Principles 


Introduction to Microeconomics 
Macroeconomics Principles 
Principles of Macroeconomics 
Principles of Macroeconomics 
Principles of Microeconomics 
Macroeconomics 

Principles of Microeconomics 
Principles of Microeconomics 
Principles of Macroeconomics 


Introduction to Statistics 
Introduction to Statistics 


Introduction to Psychology 


1.00 


Favors 
Courseware 


Boldface indicatesthat the impact estimate is statistically significant (i.e., we can be confident that the difference in average grades for students in 
courseware and business-as-usual course sections did not occur by chance). In courseware implementations that are not in boldface, we cannot 
rule out the possibility that the observed difference in courseware and business-as-usual section average grades was attributable to chance. 
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One of the things we noticed in examining instructor 
survey responses is that instructors who volunteered 
to use the courseware had more positive perceptions 
of it. It seemed reasonable to conjecture that 
courseware implementations by volunteer instructors 
might be better and hence lead to stronger student 
outcomes than those by instructors who were 
required to use the courseware. When the impact 
estimates were analyzed by instructor volunteer 
status, however, we found that although impact 
estimates tended to be more positive in cases where 
courseware-using instructors had volunteered to use 
the product than when they had been required to use 
it, the difference did not attain statistical significance. 


Other instructor features examined in the meta- 
analysis included whether or not the course version 
using the courseware was taught by a tenured or 
tenure-track faculty member as opposed to an 
instructor or adjunct and whether or not the person 
teaching it was teaching full time versus part time. 
Courseware impact estimates did not vary significantly 
between faculty and other kinds of instructors. 

There was a difference, however, based on whether 
instructors using the courseware were teaching 

full time. Courseware impacts were smaller in 
studies where most of the instructors using the 
courseware were employed part time. One might 
expect this condition to be more common in two- 
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year colleges than in four-year colleges, but that was 
not the case (only 2 of the 8 studies involving mostly 
part-time instructors were conducted in two-year 
colleges). But instructor employment status did vary 
with the modality of instruction. In three-quarters of 
the studies where instructors using the courseware 
were all or mostly part-time employees, an online 
course using the courseware was being compared to 
an online course without the NGCC product. 


We also looked at whether the instructors 
implementing the NGCC courseware were using it 

for the first time as part of the implementation study. 
This was the case for all the instructors in four impact 
studies and the majority of instructors in another 
two. All of the instructors in 15 of the studies had 
implemented the courseware previously, but this did 
not result in any discernible advantages in terms of 
their students’ learning outcomes. 


The value of these moderator variable analyses is 
limited to the extent that multiple conditions co-occur 
in many of the implementation studies. For example, 
as noted above, most of the studies where courseware 
instructors were teaching part time were also cases 

of contrasting two versions of online teaching. Under 
these circumstances, we must be cautious about 
interpreting patterns in the data shown in Figures 3-6. 
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Conditions Associated with Stronger NGCC Courseware Impacts 


¢ Course taught in 4-year college 


¢ Subject is Science, Math/Statistics or Psychology 


e Instruction with courseware compared to instruction occurring primarily face-to-face 


e Class sections of more than 50 students 


¢ Courseware instructor is teaching full time. 


What implementation practices are 
associated with better courseware 
outcomes? 


In addition to looking at the conditions under which 
courseware was being used, as Summarized above, 
we also examined a number of support practices 
for courseware implementation. The practices we 
examined included receipt of training on how to 
use the courseware, receipt of training on how to 
integrate the courseware with instruction, number 
of hours of training, vendor support, campus leader 
support, support from colleagues, and campus 

IT support. Surprisingly, none of these training 

and support variables was a significant moderator 
of courseware effectiveness. Failure to find any 
practices that moderated courseware impacts 
suggests that the sample of studies in our analysis 
may have had limited variability in terms of these 


practices or that the influence of practices is relatively 


weak and can be detected only with larger samples 
of studies. 


To provide a larger set of studies for exploring 
practices as moderators of courseware effects, we 
combined data from the 28 NGCC impact studies 
with data from the 12 earlier adaptive courseware 
implementations examined by Yarnall, Means, and 
Wetzell (2016). A meta-analysis of the resulting 40 
adaptive courseware implementation studies This 
larger meta-analysis also uncovered a number of 
practices associated with more positive impacts for 
adaptive courseware implementations:”° 


e Instructors volunteering to implement the 
courseware 


e Training for instructors on how to integrate the 
courseware with one’s intended pedagogy?! 


e Campus leader support 
e Colleague or peer support 


e Prior experience teaching with the adaptive 
courseware 


20 Findings reported earlier for more positive impacts in four-year colleges, in science and psychology courses, and when shifting from face-to- 


face to blended learning held also for this larger study sample. 


21 Impacts did not vary depending on whether or not instructors had received training on how to use the product per se. 
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Practices Associated with More Positive Courseware Impacts 


¢ Working with instructors who volunteer to try out the courseware 


e Providing training for instructors on how to integrate the courseware into their practice 


e Demonstrations of support for courseware use on the part of campus leaders 


¢ Peer support from colleagues implementing the same courseware 


Are these impact findings related to 
the strength of the study design? 


It is not unusual to find stronger impacts for an 
educational intervention in those studies with a 
weaker research design. For the NGCC evaluation 
we coded each impact study in terms of a number 
of characteristics related to the strength of the 
study--namely, sample size, whether the comparison 
group was taking the course at the same time as 
the treatment group, and whether or not the same 
instructors taught both courseware and comparison 
course sections. We did not code the studies for the 
rigor of their research design per se because all of 
the impact studies were either quasi-experimental 

or observational in nature. None of them used a true 
experimental design (with students and instructors 
randomly assigned to condition), leaving open the 
possibility that observed differences were a function 
of differences between instructors and students 

in the two courseware conditions rather than the 
contribution of the courseware itself. Our analytic 
model controlled for student characteristics likely to 
be related to course outcomes (as discussed above), 
but we could not control for instructor differences. 


Among the three study variables we tested as possible 
moderators of courseware impact (sample size, 
condition timing, and instructor overlap), only the last of 
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these made a significant difference. Impact studies in 
which there was no overlap between instructors 
in the NGCC courseware and comparison 

course sections tended to find more positive 
effects. This finding raises the possibility that where 
courseware and non-courseware sections differed 
significantly, the instructors rather than the use of the 
courseware product might have been responsible. It 
also underscores the value of assigning instructors to 
courseware treatment conditions so that courseware 
impacts can be disentangled from instructor effects 
when rigorous testing of courseware impacts is needed. 


Subgroup Analyses 


Because the foundation’s strategic goal for 
postsecondary education focuses on low-income and 
under-represented minority students, we performed 
separate analyses of courseware impacts for these 
two student subgroups. (African American and Latino 
students were combined to obtain greater statistical 
power.) Some earlier research suggests that low- 
income and under-represented minority students 
perform more poorly in online and blended learning 
courses than in conventional face-to-face courses 
(Figlio, Rush, & Yin, 2013; Jaggers, 2011), which 
would lead to the expectation that these subgroups 
would do better in the BAU condition than in the 


NGCC courseware condition of impact studies. 
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However, these studies involved earlier generations of 
online learning resources without the interactive and 
adaptive elements of NGCC courseware. 


For these student subgroup impact analyses, we 
applied the same screens for minimum sample size 
and baseline equivalence (for student characteristics 
other than the one defining the subgroup) as we 

did for the analyses described above for the total 
student sample. For low-income students, seven 

of the datasets met the sample size and baseline 
equivalency criteria; ten of the under-represented 
student datasets met these criteria. 


Figure 7 shows the courseware effect estimates for 
low-income students, and Figure 8 shows estimates 
for under-represented minority students. 


Although the average effect size for low-income 
students’ grades (+0.10) did not attain statistical 
significance, it is on a par with the effect size for 
students overall (+.09) and it is clear that the use of 
courseware does not generally put low-income students 
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at a disadvantage. There was only one case among the 
seven studies in this meta-analysis in which low-income 
students earned better grades in the BAU version of 
the course. In two cases, low-income students earned 
significantly better grades in the NGCC version of the 
course, and in four cases there was no significant effect 
of course version on low-income students’ grades. 


The results for under-represented minority students 
(African American or Latino) provide further indication 
that courseware use does not have inverse impacts 
for the students who are the focus of the foundation’s 
postsecondary strategy. The average courseware 
impact on under-represented students’ grades 
was an improvement of +0.16 standard deviation 
units, which was statistically significant. In the 
ten individual studies of course outcomes for under- 
represented minority students, five found significantly 
better outcomes in the courseware condition 

than in BAU. There was only a single courseware 
implementation where minority students fared worse 
in the courseware condition than with business 


Figure 7. NGCC Courseware Impacts for Low-Income Students 


Institution Name 


Course 


Southern U and A&M ob General Biology 
College of Canyons ———_fi-+#——_ Introduction to Statistics 
Tidewater —ii- Introduction to Business 
Mohave r— + Biology Concepts 
Southern U and A&M Ss al General Biology 
UMUC —_ }— Introduction to Statistics 
UMUC —[ ++ Principles of Macroeconomics 
UMUC +— +-— Marketing Principles 
Total > 
-1.00 -0.50 0.00 0.50 1.00 
avors Favors 
Business-as-Usual Courseware 


Boldface indicates that the impact estimate is statistically significant (i.e., we can be confident that the difference in average grades for students in 
courseware and business-as-usual course sections did not occur by chance). In courseware implementations that are not in boldface, we cannot 
rule out the possibility that the observed difference in courseware and business-as-usual section average grades was attributable to chance. 
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as usual (the same implementation for which low- The number of impact studies for which we could 
income students did worse with the courseware). In do these subgroup analyses was rather small, but 
four datasets student outcomes were equivalent in these results are encouraging. Figure 9 summarizes 
the two versions of the course. the impact findings for low-income and under- 


represented minority students. 


Figure 8. NGCC Courseware Impacts for Under-represented Minority Students 


Institution Name Course 
Miami-Dade —— General Education Biology 
College of Canyons —_—_ Introduction to Statistics 
Southern U and A&M + General Biology 
National Louis U | Survey of Bioscience 
UMUC —E- Introduction to Business & Mgmt. 
Tidewater —h Introduction to Business 
WGU (Full-time only) ti Introduction to Psychology 
Southern U and A&M _t-— General Biology 
UMUC —H- Principles of Macroeconomics 
UMUC +f Marketing Principles 
Total a> 
-1.00 -0.50 0.00 0.50 1.00 
Favors Favors 
Business-as-Usual Courseware 


Boldface indicates that the impact estimate is statistically significant (i.e., we can be confident that the difference in average grades for students in 
courseware and business-as-usual course sections did not occur by chance). In courseware implementations that are not in boldface, we cannot 
rule out the possibility that the observed difference in courseware and business-as-usual section average grades was attributable to chance. 


Figure 9. NGCC Impacts on Grades for Low-Income and Under-represented 
Minority Students 
Low-income Students Under-represented minority students 


Od 


| No Impact ma Negative Impact S Positive Impact 
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Instructor Perceptions 
of NGCC Courseware 


Courseware impacts are not the only factor 
influencing market reactions to these educational 
resources, of course. Ease of use, availability of 
implementation supports, and perceived value 

are additional considerations. To capture these 
dimensions, SRI administered an online survey to all 
instructors who used an NGCC courseware product 
with one or more of their classes in fall 2016. To 
identify the survey sample, SRI asked the grantees 
to provide a list of names and email addresses for all 
instructors using their courseware that term. 


The online NGCC Instructor Survey addressed topics 
concerning courseware use at the classroom level, 
such as how the courseware was implemented, 
supports sought and received for courseware 
implementation, perceptions of the courseware, and 
future plans for using the courseware or recommending 
it to others. To support analyses of the courseware’s 
cost effectiveness, additional survey items asked about 
the amount of time the instructor spent on various 
functions related to preparing for and implementing 
instruction with the courseware and for comparisons 
of these time estimates to those for the same functions 
without using adaptive courseware. 


After fall 2016, SRI surveyed instructors using NGCC 
products in subsequent terms (spring or fall 2017) 

if data from that term was to be used in a study of 
the product’s impact on student learning. In all, 550 
instructors out of the 658 from whom SRI requested 
surveys, submitted responses, for a response rate of 
78%. Survey responses for each grantee’s instructors 
using products in fall 2016 or in 2017 are shown in 
Appendix B. Below we highlight selected instructor 
survey responses aggregated across all grantees. 


Next Generation Courseware Challenge Evaluation 


SRI Education 


A majority of surveyed NGCC instructors (70%) 
had a choice in whether or not to use the 


courseware in their classes. The overwhelming 
majority of instructors reported that their motivations 
for trying out the courseware included the expectation 
that it would enhance student engagement (87%), 
their desire to increase the amount of individualized 
instruction in their course (84%), and their interest in 
exploring online teaching and learning (72%). 


A majority of instructors felt adequately or well 
prepared to use their NGCC courseware (69%). 
Most instructors had participated in some form of 
formal training for using the courseware (78%), and 
a majority also received some kind of support other 
than formal training (56%). Of the 56%, nearly all 
received support from courseware technical and 
support staff (85%), and slightly over half (51%) 
received support from colleagues who were also 
using the courseware. Both formal training and 
additional support were on average perceived as 
being helpful and satisfying. 


Most instructors who used NGCC courseware had 
experience with online or blended learning prior 

to using the NGCC product (67%). Instructors 
reported spending an average of approximately 
8.37 hours learning to use the courseware and 
16.24 hours integrating the courseware into 
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their courses, though the range of reported hours 
was very wide, with some instructors reporting 
spending zero hours while others reported spending 
40 or more hours. 


Roughly a third (32%) of instructors described 
the NGCC courseware as a core component of 
their course that substituted for a printed text. 
This practice has potential to reduce the course 
costs incurred by students. One out of seven NGCC 
instructors (14%) said they used the courseware in 

a fully online course. Just 9% of instructors reported 
that they used the courseware in a “flipped” classroom 
format. The most frequently used instructor-facing 
courseware features were: the online gradebook, 
dashboards of class-level and student-level progress, 
and making class materials available to students. 


After implementing the courseware, a majority 

of instructors felt the NGCC courseware 

had improved teaching and learning in their 
course. In terms of how courseware brought about 
improvements, over half of instructors agreed that the 
courseware provided the following benefits for their 
instructional practice: 


© supporting students in deeper learning (63%), 


e increasing student engagement with course 
content (63%), 


e better monitoring of individual student progress 
(57%), 


e improved tracking of whole-class progress 
(53%), and 


© presenting content more effectively (52%). 
Most instructors agreed that using the 


courseware provided the following benefits for 
their students’ learning: 


e better understanding and remembering of course 
content (66%), 
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e staying more engaged with the course overall 
(64%), and 


e better analysis and application of course content 
(62%). 


Overall, the great majority of NGCC instructors 
reported being moderately or highly satisfied 
with the courseware (88%). Eighty percent said 
they planned to use the NGCC product in their 
course in the future. Instructors said they were 
somewhat likely to recommend the courseware they 
used to a friend or colleague teaching the same 
course (average response 6.52 on a scale from 0 = 
Extremely Unlikely to 10 = Extremely Likely with 5= 
Neither Likely nor Unlikely). 


SRI researchers’ visits to pilot implementation sites 
where they could observe courseware in use and 
interview teachers and students revealed aspects of 
the NGCC courseware that were viewed positively: 


Both instructors and students liked the interactive 
elements of the courseware; they found them 
engaging and believed that they were conducive to 
deeper learning. 


Students also liked having formative assessments 
to check their understanding, provided that the 
assessments were perceived as relevant to preparing 
them for course tests and grades. 


Instructors liked the use of adaptive formative 
assessment activities and hints. Students liked them 
without necessarily being aware they were adaptive. 


At several sites, instructors said they thought the 
courseware was increasing the time they could 
spend interacting with students. Some described 
flipping the classroom, and others said it reduced the 
time they had to spend grading and justifying their 
grades to disgruntled students. 
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Courseware Cost Study 
and Cost Effectiveness 


To make good decisions about implementing 
particular instructional interventions, educators 
need to understand the likely costs as well as the 
likely benefits. Cost is a practical consideration 

that can dramatically shape how knowledge 

about effective practices is translated into action 

in districts. Reliance on effectiveness alone may 
encourage adoption of interventions that are too 
expensive to sustain with fidelity (Bakia, Caspary, 
Wang, Dieterle, & Lee, 2011; Harris, 2009; Hollands 
et al., 2014). Analytic approaches that examine 
costs are relevant in the context of rising prices 

in education and decreasing educational budgets 
(Bowen, 2013; Hollands et al., 2014). In addition to 
questions of impact and efficiency, policy-makers 
and administrators require information related to 
affordability to address basic questions such as: Will 


a new program or approach increase costs, and if so, 


by how much? While cost-analyses of educational 
interventions are not yet common, studies of cost 
and cost-effectiveness are readily available in the 

health and human services sector. 


The set of NGCC implementations for which we are 
conducting cost and cost effectiveness analyses is 
limited to those combinations of grantee products, 
sites, and courses for which we have usable impact 
study data. A cost model was developed for nine 
fall 2016 impact datasets involving products from 
four grantees (CogBooks, Lumen, OLI, and Smart 
Sparrow). 
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Data for the cost analyses were collected primarily 


from: 
e NGCC instructor surveys 


e administrative and publicly available data 
regarding salaries and material costs 


e interviews with instructors and administrators at 
selected sites 


e implementation information gathered from 
grantees by their SRI liaisons 


For each cost analysis all relevant data were entered 
into a template that captured possible cost impacts 
for setting up and delivering courses with and 
without the courseware, both in the initial year and 
on an ongoing basis. Following Levin et al. (2017), 
we took an “ingredients” approach, identifying 

all inputs regardless of whether administrators, 
faculty members, or students bore the costs. 

These categories included the costs of providing 
instruction (facilities and technology, staff professional 
development, administrative and technical support, 
course restructuring and system setup, differences in 
staff teaching time), and differences in access costs 
for students (subscription fees, textbook costs). Staff 
labor estimates took into account the staff role and 
salary levels (e.g., faculty members versus teaching 
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assistants). Cost estimates were calculated for the 
academic term for which course impact data were 
collected, as well as any expenses related to initial 
product set up and integration. 


The results of cost effectiveness analyses are often 
depicted in what is called a cost effectiveness plane, 
such as that shown in Figure 10. Each quadrant in the 
plane represents one of the four possible combinations 
of relative cost and relative effectiveness. Specifically, 
an intervention could be (1) less expensive but also less 
effective than the alternative, (2) less expensive and 
more effective, (3) more expensive and more effective, 
or (4) more expensive and less effective relative to the 
alternative. Clearly, an intervention falling in quadrant 


Figure 10. The Cost Effectiveness Plane 
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2 (more effective and less costly) would be the ideal 
case, and one falling into quadrant 3 (more expensive 
and less effective) is least desirable. Often, however, 
the intervention being evaluated is not better on both 
the cost and the effectiveness dimensions, making 

it more difficult to decide what to do on the basis 

of cost effectiveness analyses. It is common for an 
intervention to fall into quadrant 3 (more effective and 
more costly than the usual approach), and there are 
also interventions that are less effective but less costly 
(quadrant 1). Whether or not decisionmakers want 

to pursue interventions falling into either of these two 
quadrants depends on the relative value they place on 
cost savings versus effectiveness (Levin et al., 2017). 


Cost (Treatment - Comparison) 


f@lUETele-laja- 
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More Effective 


Quadrant 2 


Less Costly 
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Table 10 shows the incremental costs/savings 
associated with using courseware in the nine 

NGCC cost effectiveness analyses alongside the 
corresponding impact on course grades (from Table 8). 


Table 10. Cost Effectiveness Analyses: Effect Size and Cost Difference 


Courseware Institution and Course Effect size (g) Cost 
CogBooks aay a +0.23 $47 
CogBooks ae PE COME +0.38 $228 
CogBooks Scat eara and A&M, General 40.46 $224 
Lumen Uae watey Community Coleus) +0.16 $167 
Lumen ew atone oma Colca 0.89 $79 
Stanford OL Ene awe ee 40.30 ($44) 
Stanford OL nase et WeeCal olny +0.30 $176 
Smart Sparrow Ee Sala) Ce lieces +0,.22 $51 
Smart Sparrow bi Soo aie AOA $36 
Average +0.20 $107.11 
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Figure 11 shows the nine NGCC cost effectiveness 
analysis results arrayed on the cost effectiveness 
plane.” As the figure shows, 7 of the 9 analyses fell 
into quadrant 2—courseware was more effective 
and less costly than business as usual. One result 
fell into quadrant 3 (more effective and more costly) 
and one into quadrant 4 (less effective and less 
costly). Thus, in 8 out of 9 cases, courseware use 
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led to cost savings. These savings stemmed primarily 
from avoidance of textbook costs for students. 
Importantly, these cost savings were not achieved 

by compromising student learning. As already 

noted, students in the courseware condition earned 
higher course grades than those in the business- 
as-usual version of the course in 8 of these 9 cost 
effectiveness studies. 


Figure 11. NGCC Studies on the Cost Effectiveness Plane 


Cost Difference 


$100.00 


$50.00 


-0.5 


Stanford OLI: +0.3, $44.00 
e 


Effect Size 


0.1 0.2 0.3 0.4 0.5 


Smart Sparrow: +0.17, ($36.00) 


Lumen: -0.39, ($79.00) 
e) 


($400.00) 
($150.00) 
($200.00) 


($250.00) 


PE oaBooks: +0.23, ($47.00) 
Smart Sparrow: +0.22, ($51.00) 


Lumen: +0.16, ($167.00) 


e 
e 
Stanford OLI: +0.3, ($176.00) 


CogBooks: +0.46, ($224.00) 
@ 


® 
CogBooks: +0.38, ($228.00) 


22 The student sample sizes and semesters of data used for the cost analysis are, in some cases, different than those reported in the overall effect 
size and other analyses reported in the previous sections of this report. 
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We cannot make strong claims about the 
generalizability of these nine cases. We had 
adequate data for estimating costs for only 9 out 

of the 28 NGCC courseware implementations 

with efficacy estimates, and this sample is not 
necessarily representative of all NGCC courseware 
implementations yet alone of courseware use in 
general. Moreover, cost estimates are very sensitive 
to the amount of time the instructor spends preparing 
for and delivering the course in both treatment and 
comparison conditions, and instructor practices 
vary markedly. But we can say that the preliminary 
evidence available to us suggests that high-quality 
courseware can be cost effective. Our analyses 
show that if the initial startup costs of implementing 
courseware are amortized over the number of 
terms for which the institution expects to use the 
courseware and costs to students are factored in, 
courseware can contribute to increased efficiency 
in higher education. For the nine NGCC cases for 
which we had adequate data for a cost-effectiveness 
analysis, use of NGCC courseware moved the 
average student’s performance from the 50th to the 
58th percentile while saving approximately $107 
per student. Cost details for the nine individual cost 
analyses appear on the pages that follow. 
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Courseware Cost Case Studies 
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CogBooks in Arizona State University Biology 100 


¢ Term: Fall 2016 ¢ Business as Usual (BAU) Comparison: 


SGAuiee FIO 0G Lecture course with textbook in fall 2014 


e Number of students considered in the cost 
effectiveness analysis: 437 


¢ Impact estimate for effect on course 
grade: 0.23 


e Average section sizes: 66 treatment, 201 BAU 


Cost Components 


Cost Cost 
Analysis Level Cost Element Difference/ Difference/ 
Institution Student 
Student-level Material (Software vs. textbook) -$70 
Equipment (e.g. classroom renovation, computers) $0 $0 
Instructional design 
(e.g. instructor labor, instructional design staff) $8,400 $19 
Instructor- Instructor professional development and training $1,260 $3 
level (including TAs) ; 
Teaching time difference (e.g. preparation, instructional $0 $0 
time, grading) 
Other campus staff (e.g. technical/administrative support) $500 $1 
Communication (e.g. within campus, with vendors) $0 $0 
-$47 


Overall 


Material (software vs. textbook): 
e The largest cost difference between the courseware and BAU version of the course was the student 
subscription fee associated with CogBooks in relation to the BAU textbook. This provided a savings of 


$70 per student. 


Equipment (e.g. classroom renovation, computers): 

e No facilities costs were incurred specifically for use of CogBooks at ASU. ASU taught the CogBooks BIO 
100 course in a room built for active learning, which included a flat floor (vs. tiered seating), round tables 
for 6, seating for 75 people, white boards for each table, high volume Wi-Fi, and an audiovisual system 
with projector and microphones. ASU would have built this room regardless of its use of CogBooks, so 
this room was not considered a CogBooks-related expense for the courseware implementation. 


e Students were required to bring their own laptops so there were no associated computer costs. 
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CogBooks in Arizona State University Biology 100 (cnt'a) 


Instructional design (e.g. instructor labor, instructional design staff): 

e Administrator time and Instructional designer supporting the integration were provided in the preparation 
phase for CogBooks usage in the biology course. After initial implementation, ASU incurred no further 
ongoing costs for administrator or instructional designer support. 

e Administrator spent time supporting the integration and use of CogBooks in the preparation phase. ASU 
incurred no ongoing costs for administrative support after the initial implementation. 


e Two faculty members were given release time by their department to work on integrating the BIO 100 
course materials into CogBooks. This included reviewing and editing the open-source textbook, rewriting 
exams, and designing active learning lessons. Three other faculty members provided validation work of 
the CogBooks BIO 100 course, reviewing content, correcting errors, and revising materials. 


Instructor professional development and training (including TAs): 
e Initial professional development was provided for 4 faculty members for the fall 2016 semester. 


e Ona yearly basis, Teaching Assistants (TAs) participate in a training workshop. 


Teaching time difference (e.g. preparation, instructional time, grading): 
e For faculty members, CogBooks did not save or cost additional instructional time overall. 


Other campus staff (e.g. technical/administrative support): 
e On-campus technology support was provided in the preparation phase and the first semester of CogBooks use. 
ASU incurred no ongoing costs for on-campus technology support after initial implementation. 
Cost Effectiveness 


Implementation of CogBooks at ASU in fall 2016 lowered costs and provided better academic results 
compared to business as usual in a prior term. 


More Effective 


LOfele | sfore) 
BIO 100 


Less Costly 
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CogBooks in Southern University and A&M College 
General Biology 104 


¢ Term: Fall 2016 ¢ Business as Usual (BAU) Comparison: 
Course RIO 104 Lecture course with textbook in spring 2017 


ONGibonortradcnectcane dered mmclcoce ¢ Impact estimate for effect on course grade: 0.38 


effectiveness analysis: 283 e Average section sizes: 56 treatment, 62 BAU 


Cost Components 


Cost Cost 
Analysis Level Cost Element Difference/ Difference/ 
Institution Student 

Student-level Material (software vs. textbook) -$236 

Equipment (e.g. classroom renovation, computers) $0 $0 

Instructional design 

(e.g. instructor labor, instructional design staff) $1,365 $5 
Instructor- Instructor professional development and training $0 $0 
level (including TAs) 

Teaching time difference (e.g. preparation, instructional $478 $2 

time, grading) 

Other campus staff (e.g. technical/administrative support) $0 $0 

Communication (e.g. within campus, with vendors) $328 $4 
Overall -$228 


Material (software vs. textbook): 
e The largest cost difference between the courseware and BAU version of the course was the student 
subscription fee associated with CogBooks in relation to the BAU textbook. This provided a savings of 
$236 per student. 


Equipment (e.g. classroom renovation, computers): 
e Students were required to bring their own laptops so there were no associated computer costs. No 
facilities costs were incurred specifically for use of CogBooks at Southern University and A&M College 
since computer labs and other online learning resources were already available. 
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SRI Education 


CogBooks in Southern University and A&M College 
General Biology 104 (ont'a) 


Instructional design (e.g. instructor labor, instructional design staff): 
e The BIO 104 course coordinator and 2 other lead instructors initially spent time integrating the course 
content into CogBooks. This included reorganizing the course content and rewriting the course 
PowerPoint slides. 


Instructor professional development and training (including TAs): 
e Not affecting the cost of the treatment condition, were professional development supports for faculty, 
since these were similar across the CogBooks and BAU conditions. 


Teaching time difference (e.g. preparation, instructional time, grading): 
e Increased time instructors spent in course preparation and working with students. 


Communication (e.g. within campus, with vendors): 
e Communication and problem solving between course faculty and the product developer, for example over 
issues sending email to their students was reported in the first semester of product use. 
Cost Effectiveness 


Implementation of CogBooks at Southern University and A&M College in spring 2017 lowered costs and 
provided better academic results compared to business as usual in the same term. 


More Effective 
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SRI Education 


CogBooks in Southern University and A&M College 
General Biology 105 


¢ Term: Fall 2016 ¢ Business as Usual (BAU) Comparison: 


eourse Ri@nes Lecture course with textbook in fall 2016 


« Number oretaudents considered in thecost ¢ Impact estimate for effect on course grade: 0.46 


effectiveness analysis: 224 e Average section sizes: 66 treatment, 73 BAU 


Cost Components 


Cost Cost 
Analysis Level Cost Element Difference/ Difference/ 
Institution Student 

Student-level Material (software vs. textbook) -$236 

Equipment (e.g. classroom renovation, computers) $0 $0 

Instructional design 

(e.g. instructor labor, instructional design staff) val Tas $8 
Instructor- Instructor professional development and training $0 $0 
level (including TAs) 

Teaching time difference (e.g. preparation, instructional $637 $3 

time, grading) 

Other campus staff (e.g. technical/administrative support) $0 $0 

Communication (e.g. within campus, with vendors) $418 $2 
Overall -$224 


Material (software vs. textbook): 
e The largest cost difference between the courseware and BAU version of the course was the student 
subscription fee associated with CogBooks in relation to the BAU textbook). This provided a savings of 
$236 per student. 


Equipment (e.g. classroom renovation, computers): 
e Students were required to bring their own laptops so there were no associated computer costs. No 
facilities costs were incurred specifically for use of CogBooks at Southern University and A&M College 
since computer labs and other online learning resources were already available. 
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SRI Education 


CogBooks in Southern University and A&M College 
General Biology 105 (ont'a) 


Instructional design (e.g. instructor labor, instructional design staff): 
e The BIO 105 course coordinator and 2 other lead instructors initially spent time integrating the course 
content into CogBooks. This included reorganizing the course content and rewriting the course 
PowerPoint slides. 


Instructor professional development and training (including TAs): 
e Not effecting the cost of the treatment condition were professional development supports for faculty, since 
these were similar across the CogBooks and BAU conditions. 


Teaching time difference (e.g. preparation, instructional time, grading): 
e Increased time instructors spent in course preparation and working with students. 


e Communication (e.g. within campus, with vendors): 


e Communication and problem solving between course faculty and the product developer, for example over 
issues sending email to their students was reported in the first semester of product use. 


Cost Effectiveness 


Implementation of CogBooks at Southern University and A&M College in spring 2017 lowered costs and 
provided better academic results compared to business as usual in the same term. 


11 (0) = yp (107 11142) 


Ofele] sTole) < 
BIO 105 


| XS OF 0k 1114 


Next Generation Courseware Challenge Evaluation: Case Studies 58 


SRI Education 


Lumen Learning in Tidewater Community College 
Business 100 


¢ Term: Fall 2016 e Business as Usual (BAU) Comparison: 
eGaurs= BUSAO0 Regular online course without Lumen 


: ; software in fall 2016 
e Number of students considered in the cost 


effectiveness analysis: 349 e Impact analysis effect size: 0.16 


e Average section sizes: 24 treatment, 21 BAU 


Cost Components 


Cost Cost 
Analysis Level Cost Element Difference/ Difference/ 
Institution Student 

Student-level Material (software vs. textbook) -$160 

Equipment (e.g. classroom renovation, computers) $0 $0 

Instructional design 

(e.g. instructor labor, instructional design staff) $1,905 Bae 
Instructor- Instructor professional development and training $520 $1.49 
level (including TAs) : 

Teaching time difference (e.g. preparation, instructional -$5,611 $16 

time, grading) , 

Other campus staff (e.g. technical/administrative support) $24 $0.06 

Communication (e.g. within campus, with vendors) $395 Gide 
Overall -$167 


Material (software vs. textbook): 
e The largest cost difference was the student subscription fee associated with Lumen Waymaker in relation 
to the BAU textbook for one semester. This provided a savings of $160 per student. 


Equipment (e.g. classroom renovation, computers): 
e No equipment or facilities costs were incurred specifically for use of Lumen Waymaker at TCC. There were 
no differential costs with respect to equipment and facilities; sections that used Lumen Waymaker did not 
require any additional facilities or equipment beyond that which BAU sections used. 
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SRI Education 


Lumen Learning in Tidewater Community College 
Business 100 (ont'a) 


Instructional design (e.g. instructor labor, instructional design staff): 

e Within the Business department, an administrator and several faculty members consulted on the 
course redesign, making it possible to use Lumen Waymaker in all sections of the course. In addition, 
the administrator and several faculty members spent time supporting the ongoing integration and 
use of Lumen Waymaker in fall 2016. After initial implementation, TCC incurred no ongoing costs for 
administrator or faculty support of course redesign and integration of Lumen Waymaker. 


e Faculty were offered stipends to pilot the use of Lumen Waymaker in BUS 100 for the fall 2015 semester. 


Instructor professional development and training (including TAs): 
e Initial professional development was provided for one administrator and faculty. After initial implementation, 
TCC incurred no ongoing costs for professional development for either the administrator or faculty. 


Teaching time difference (e.g. preparation, instructional time, grading): 
e For faculty members, Lumen Waymaker saved instructional time overall for all instructors. The primary 
driver for time saving was that instructors were provided with a “master course” shell that reduced pre- 
course preparation time. 


Other campus staff (e.g. technical/administrative support): 
e Minimal technology support was provided in the preparation phase, from both the Virginia Community College 
System level and TCC Campus IT. TCC incurred no ongoing technology support costs after initial implementation. 


Communication (e.g. within campus, with vendors): 
e Administrators and senior leadership spent time in communication with the vendor during the initial fall 
2016 semester. TCC incurred minimal ongoing costs associated with communication with the vendor after 
the initial implementation. 


Cost Effectiveness 


Using Lumen Waymaker at TCC in fall 2016 lowered costs relative to BAU while providing better academic results. 


More Effective 


Lumen Waymaker 
BUS 100 


Less Costly 
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SRI Education 


Lumen Learning in Tidewater Community College 
Economics 201 


¢ Term: Fall 2016 ¢ Business as Usual (BAU) Comparison: 
Regular online course without Lumen 
software in fall 2016 


¢ Number of students considered in the cost 
effectiveness analysis: 46 
¢ Impact analysis effect size: -0.39 


e Average section sizes: 21 treatment, 29 BAU 


Cost Components 


Cost Cost 
Analysis Level Cost Element Difference/ Difference/ 
Institution Student 
Student-level —_ Material (software vs. textbook) -$190 
Equipment (e.g. classroom renovation, computers) $0 $0 


Instructional design 


(e.g. instructor labor, instructional design staff) $4,036 $67 
Instructor- Instructor professional development and training $41,788 $30 
level (including TAs) : 

Teaching time difference (e.g. preparation, instructional $250 $4 

time, grading) 

Other campus staff (e.g. technical/administrative support) $84 $1 

Communication (e.g. within campus, with vendors) $455 $8 
Overall -$79 


Material (software vs. textbook): 
e The largest cost difference was the student subscription fee associated with Lumen Waymaker relative to 
the cost of the BAU textbook for one semester. This provided a savings of $189.50 per student. 


Equipment (e.g. classroom renovation, computers): 
e No equipment or facilities costs were incurred specifically for use of Lumen Waymaker at TCC. There were 
no differential costs with respect to equipment and facilities. Sections that used Lumen Waymaker did not 
require any additional facilities or equipment beyond that which BAU sections used. 
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SRI Education 


Lumen Learning in Tidewater Community College 
Economics 201 cont'd) 


Instructional design (e.g. instructor labor, instructional design staff): 
e Faculty spent time in course redesign and the integration of Lumen Waymaker into the economics course, 
as well as supporting ongoing integration and use of Lumen Waymaker in the initial semester of Fall 2016. 


e Lumen Waymaker staff spent time modifying instructional materials and assessments in response to 
faculty feedback during the fall 2016 semester. 


Instructor professional development and training (including TAs): 
e Initial professional development was provided for one administrator and the participating faculty member. After initial 
implementation, TCC incurred no ongoing costs for professional development for either the administrator or faculty. 


Teaching time difference (e.g. preparation, instructional time, grading): 
e One faculty member was provided a stipend to pilot the use of Lumen Waymaker in ECO 201 for the fall 
2016 semester only. 


Other campus staff (e.g. technical/administrative support): 
e Minimal technology support was provided in the preparation phase, either by the Virginia Community College 
System or by TCC Campus IT. TCC incurred no ongoing technology costs after initial implementation. 


Communication (e.g. within campus, with vendors): 
e Administrators and senior leadership spent time in communication with the vendor during the initial fall 
2016 semester. TCC incurred minimal ongoing costs associated with communication with the vendor after 
the initial implementation. 


Cost Effectiveness 


Implementing Lumen Waymaker in the TCC economics course in fall 2016 lowered costs but resulted in lower 
course grades compared to BAU. 


11 (0) = my p (=e 1112) 


| XS OF 0k 1114 


Next Generation Courseware Challenge Evaluation: Case Studies 62 


SRI Education 


Stanford OLI in University of Maryland-University 
College Statistics 200 


¢ Term: Fall 2016 ¢ Business as Usual (BAU) Comparison: 
Regular UMUC online course without 


e Course: STAT 200 
Stanford/OLI software* 


e Number of students considered in the cost 


effectiveness analysis: 60 ¢ Impact estimate for effect on course grade: 0.30 


e Average section sizes: 30 treatment, 30 BAU 


* An academic term at UMUC is 8 weeks long, rather than the typical 16 weeks at conventional colleges. During fall 2016 each STAT 
200 instructor taught two 8-week terms. We selected an 8-week term as our length of intervention, and used the number of students 
that would enroll in one academic term. 


Cost Components 


Cost Cost 
Analysis Level Cost Element Difference/ Difference/ 
Institution Student 

Student-level Material (software vs. textbook) $15 

Equipment (e.g. classroom renovation, computers) $0 $0 

Instructional design 

(e.g. instructor labor, instructional design staff) a oe 
Instructor- Instructor professional development and training $73 $1 
level (including TAs) 

Teaching time difference (e.g. preparation, instructional $981 $16 

time, grading) 

Other campus staff (e.g. technical/administrative support) $420 $7 

Communication (e.g. within campus, with vendors) $0 $0 
Overall $44 


Material (software vs. textbook): 
e The largest cost difference was the institutional subscription fee associated with Stanford/OLI ($14,000 
one-time fee). UMUC does not pass courseware fees along to students; there is no textbook/software fee 
for students in either condition. 


Equipment (e.g. classroom renovation, computers): 
e No equipment or facilities costs were incurred specifically for use of Stanford/OLI at UMUC. There were no 
differential costs with respect to equipment and facilities. Sections that used Stanford/OL! did not require 
any additional facilities or equipment beyond that which BAU sections used. 
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SRI Education 


Stanford OLI in University of Maryland-University 
College Statistics 200 (ont'a) 


Instructional design (e.g. instructor labor, instructional design staff): 
e ALearning Design & Solution team spent time preparing the course for the first implementation. UMUC’s 
Content Operations also team spent time supporting the Stanford/OLI courses compared to BAU courses 
in the preparation phase. 


e A department head and two faculty members were involved in course redesign and integrating Stanford/OLl into 
the statistics course. The department head's time supporting the initial integration and use of Stanford/OL! was the 
same amount of time that would be spent supporting BAU courses. After initial implementation, UMUC incurred no 


ongoing costs for department head or faculty support of course redesign and integration of Stanford/OLl. 


Instructor professional development and training (including TAs): 
e Initial professional development was provided for the department, for both tenured faculty and adjunct faculty. 


Teaching time difference (e.g. preparation, instructional time, grading): 
e For faculty members, Stanford/OL! implementation required additional instructional time each term. The primary driver 
for this time requirement was instructors’ perception that the content was not at a level appropriate for their introductory 
statistics classes, and so spent approximately an hour per week explaining the more advanced problems with students. 


e Faculty stipends were provided for faculty who participated in the pilot fall 2016 semester. 
Communication (e.g. within campus, with vendors): 

e The department head and UMUC senior leadership spent time time in communicating with the OLI vendor 
initially. After the initial implementation, UMUC did not experience significant ongoing costs associated 
with communication with the vendor. 

Cost Effectiveness 


Implementation of Stanford OL! in fall 2016 raised costs but provided better academic results compared to 
business as usual in a prior term. 
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SRI Education 


Stanford OLI in College of the Canyons 
Mathematics 140 


¢ Term: Fall 2016 ¢ Business as Usual (BAU) Comparison: 
© Course: MATH 140 Lecture course with textbook in spring 2017 


* Number ot etudents considered in thecost ¢ Impact estimate for effect on course grade: 0.30 


effectiveness analysis: 498 e Average section sizes: 34 treatment, 32 BAU 


Cost Components 


Cost Cost 
Analysis Level Cost Element Difference/ Difference/ 
Institution Student 
Student-level Material (Software vs. textbook) -$178 
Equipment (e.g. classroom renovation, computers) $0 $0 
Instructional design $0 $0 
(e.g. instructor labor, instructional design staff) 
Instructor- Instructor professional development and training $1,014 $2 
level (including TAs) , 
Teaching time difference (e.g. preparation, instructional $0 $0 
time, grading) 
Other campus staff (e.g. technical/administrative support) $0 $0 
Communication (e.g. within campus, with vendors) $0 $0 
Overall -$176 


Material (software vs. textbook): 
e The largest cost difference between the courseware and BAU version of the course was the student 
subscription fee versus the textbook. College of the Canyons paid $14,000 initially to use OLI. Divided by 
16 semesters (8 years) and 498 students per semester, this fee comes to $2 per student. Students in the 
BAU condition purchased a textbook for $180, so the savings was $178. 
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SRI Education 


Stanford OLI in College of the Canyons 
Mathematics 140 (ont'a) 


Equipment (e.g. classroom renovation, computers): 
e Students used their own laptops so there were no associated computer costs. No facilities costs were 
incurred specifically for use of OLI at College of the Canyons since computer labs and other online 
learning resources were already available. 


Instructor professional development and training (including TAs): 
e Trainings for OLI were provided to administrators and teaching staff using OLI. Customizing the course to 
the OLI product also involved instructor time. 
Cost Effectiveness 


Implementation of OLI at College of the Canyons in fall 2016 lowered costs and provided better academic 
results compared to business as usual in the same term. 


More Effective 


OLI MATH 140 
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SRI Education 


Smart Sparrow in Mohave Community College 
Biology 100 


¢ Term: Fall 2016 ¢ Business as Usual (BAU) Comparison: 


wcourse BIG 100 Lecture course with textbook in fall 2016 


¢ Number of students considered in the cost 
effectiveness analysis: 106 e Average section sizes: 37 treatment, 22 BAU 


¢ Impact estimate for effect on course grade: 0.22 


Cost Components 


Cost Cost 
Analysis Level Cost Element Difference/ Difference/ 
Institution Student 
Student-level Material (software vs. textbook) -$65 
Equipment (e.g. classroom renovation, computers) $0 0 
Instructional design $17 0 
(e.g. instructor labor, instructional design staff) 
Instructor- Instructor professional development and training $126 1 
level (including TAs) 
Teaching time difference (e.g. preparation, instructional $705 7 
time, grading) 
Other campus staff (e.g. technical/administrative support) $0 @) 
Communication (e.g. within campus, with vendors) $660 6 
Overall -$51 


Material (software vs. textbook): 
e The largest cost difference between the courseware and BAU version of the course was the student 


subscription fee versus the textbook. This created a savings of $65 per student. 


Equipment (e.g. classroom renovation, computers): 
e Students used their own laptops so there were no associated computer costs. No facilities costs were 
incurred specifically for use of Smart Sparrow at Mohave Community College since computer labs and 


other online learning resources were already available. 
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SRI Education 


Smart Sparrow in Mohave Community College 
Biology 100 (cont'd.) 


Instructional design (e.g. instructor labor, instructional design staff): 
e Additional money was spent in the time invested by administrators and other college faculty related to the 


course reconstruction initially. 


e Increased instructor time included the work of redesigning the homework assignments used in the course, 
faculty training for Smart Sparrow, and time spent answering student questions and helping students use 
the product. 


Instructor professional development and training (including TAs): 
e Trainings for Smart Sparrow were provided to administrators and teaching staff using the product, as well 
as ongoing training and coaching. 


Communication (e.g. within campus, with vendors): 
e Time was spent by college staff in communication regarding the NGC project, such as course design and 
data collection. 
Cost Effectiveness 


Implementation of Smart Sparrow at Mohave Community College in fall 2016 lowered costs and provided 
better academic results compared to business as usual in the same term. 


More Effective 


Smart Sparrow 
BIO 100 


| XS Ook 1114 


Next Generation Courseware Challenge Evaluation: Case Studies 68 


SRI Education 


Smart Sparrow in National University Biology 100 


¢ Term: Fall 2016 e Business as Usual (BAU) Comparison: 
Course RIGIOO Lecture course with LaunchPad in fall 2016 


¢ Number of students considered in the cost 
effectiveness analysis: 200 e Average section sizes: 23 treatment, 22 BAU 


¢ Impact estimate for effect on course grade: 0.17 


Cost Components 


Cost Cost 
Analysis Level Cost Element Difference/ Difference/ 
Institution Student 
Student-level Material (software vs. textbook) -$65 
Equipment (e.g. classroom renovation, computers) $0 $0 
Instructional design $0 $0 
(e.g. instructor labor, instructional design staff) 
Instructor- Instructor professional development and training $5,560 $08 
level (including TAs) : 
Teaching time difference (e.g. preparation, instructional -$1,096 -$5 
time, grading) ; 
Other campus staff (e.g. technical/administrative support) $0 $0 
Communication (e.g. within campus, with vendors) $1,240 $6 
Overall -$36 


Material (software vs. textbook): 
e The largest cost difference between the courseware and BAU version of the course was the student 
subscription fee associated with Smart Sparrow in relation to the BAU product, LaunchPad. This provided 


a savings of $65 per student. 


Equipment (e.g. classroom renovation, computers): 
e Students were required to bring their own laptops so there were no associated computer costs. No 
facilities costs were incurred specifically for use of Smart Sparrow or LaunchPad since computer labs and 


other online learning resources were already available. 
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SRI Education 


Smart Sparrow in National University Biology 100 (conta) 


Instructor professional development and training (including TAs): 
e For the Smart Sparrow treatment condition, faculty were provided training and incentives to use the product. 


Teaching time difference (e.g. preparation, instructional time, grading): 
e Faculty teaching the BAU condition using LaunchPad spent time creating videos for that product. Smart 
Sparrow, for the treatment condition, did all the customization for National University, providing a savings 
in the first year of implementation. 


Communication (e.g. within campus, with vendors): 
e Time was spent by university staff in communication regarding the NGC project, inlcuding as course 
design and data collection. 
Cost Effectiveness 


Implementation of Smart Sparrow at National University in fall 2016 lowered costs and provided better 
academic results compared to business as usual in the same term. 


More Effective 


Smart Sparrow 
BIO 100 
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SRI Education 


Implications and Recommendations 


As a large, complex initiative, the Next Generation 
Courseware Challenge generated a great deal of data 
and valuable experience to support future efforts 

to leverage learning technology to close college 
attainment gaps. We close this report by highlighting 
six themes with implications for future efforts. 


When well-implemented, adaptive, next- 
generation courseware appears to be at least 
as effective as business as usual and to hold 
particular promise for enhancing course 
outcomes for under-represented students. 


Across all 28 NGCC impact studies, the average 
courseware impact on course grades was significantly 
positive but small in magnitude. More encouraging 
were the findings of analyses of courseware impacts 
for students from ethnic groups under-represented 
among college graduates. The average impact 
estimate for the ten datasets that could be used to 
evaluate the impact of NGCC courseware on under- 
represented minority students was larger than that for 
students as a whole, and half of those courseware 
implementations resulted in statistically significant 
positive impacts for under-represented minorities. 


But much remains to be learned about how 
incorporating courseware into a lower-division college 
course affects the experiences of under-represented 
students, suggesting the need for qualitative and 
longitudinal studies to complement analyses of 
course performance data. This research should 
include in-depth studies of the study behaviors 
and experiences of students from low-income 
and under-represented backgrounds in gateway 
courses with and without adaptive courseware. 


Next Generation Courseware Challenge Evaluation 


A better understanding of these students’ needs and 


experiences would inform technology developers’ 
continuing efforts to improve their courseware 
products and guidelines for implementing them. 


Efforts to scale courseware through institutional 
adoptions may sacrifice user numbers in the 
short term, but have advantages in the longer 
term. NGCC scaling data suggest that it may take 
three years for the institutional scaling strategy to attain 
parity with the direct-to-faculty strategy in terms of user 
numbers. But what these numbers do not reveal is the 
greater capacity for department-wide course redesign 
and for measuring courseware impacts in institutions 
that are undergoing wholesale adoption. To the extent 
that courseware is intended to be a stimulus for 
significant changes in college programs and pathways, 
institutional adoptions (at least program or department- 
wide) are preferable. Institutional adoptions are more 
likely to have support from institutional leaders who 
can articulate a vision for using courseware to improve 
teaching and learning and to provide resources and 
incentives (Such as faculty recognition or stipends) 

to support the effort. They also make it possible to 
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conduct more useful analyses of the impacts of course 
changes. As part of an institutional adoption, common 
learning outcomes can be defined and assessed 
across all sections of a course. It is also more likely that 
student information system data will be made available 
that can be combined with course behavior and 
outcome data in controlled studies of course redesign 
impacts. At the same time, experience suggests 

that for initial trials of new courseware products and 
teaching approaches, there are advantages in working 
with volunteer instructors. In addition to being easier 
to recruit (because there are so many more of them 
and only one person has to be convinced), individual 
instructors who are interested in trying out new 
pedagogies supported by courseware are more likely 
to use the courseware to the extent intended and to 
make connections between online and face-to-face 
components of their course. 


On balance, the NGCC experience suggests that 
providers of new courseware products may wish 

to work with individual volunteer instructors during 

the early trials of their product as they are collecting 
feedback that will help them improve the product and 
exploring different courseware implementation models. 
Once the product has achieved greater stability and 
implementation models that appear to work well have 
been articulated, it makes sense to move toward 
efforts at institutional adoption. Similarly, from the 
institutional perspective, most colleges would not want 
to discourage individual faculty members from trying 
out new digital learning tools they identify themselves, 
provided that the tools are accessible to all students 
and compatible with the campus IT infrastructure. 

But if courseware adoption is intended to be a 
stimulus for significant improvements in student 
outcomes, institutional adoptions (at least 
program or department-wide) will be needed. 
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SRI Education 


Many questions remain about the conditions 

and practices necessary for next-generation 
courseware to enhance student outcomes in 
community college settings. A particular concern 

in the NGCC student impact data is the lack of 
positive findings for implementations within community 
colleges. We know both from some of the NGCC 
community college impact studies and from past 
efforts to bring digital learning to this market (See 
Means, Peters, & Zheng, 2014) that courseware can 
improve student outcomes in community college 
courses, but clearly it does not do so consistently. 
The low number of community college courseware 
implementations with positive impacts on student 
course outcomes suggests that these institutions have 
not developed some of the organizational capacities 
and processes used in four-year colleges. At the same 
time, it should be recognized that community colleges 
are serving a different population and often do so 
primarily with part-time instructors. There is a need to 
develop models for growing the capacity to improve 
teaching and learning within community colleges and 
to document “proof of concept” cases from which 
other community colleges can learn. 


Most institutions implementing next-generation 
courseware experienced extra costs for the first 
round of implementation, but the incremental 
costs to the institution were typically modest 
and outweighed by savings experienced by 
students. Additional costs to institutions stemmed 
largely from instructors spending time learning to use 
the courseware product and planning its integration 
into their teaching. There were also modest costs 

for activities such as providing instructor training 

or IT support and time communicating with the 
courseware vendor. Most of these costs did not 
involve actual expenditures on the part of institutions, 
but rather the value of the additional staff time 
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required. The institutions in our nine NGCC cost 
studies did not attempt to save costs by increasing 
class size or reducing the number of contact hours 
when they started using courseware. These biggest 
cost savings found in our nine cases resulted from 
savings on textbook costs. An insight we gained 
from the process of collecting and analyzing cost 
data was that this is very difficult to do in cases 
where the comparison version of the course was 
taught years earlier. Instructors for those course 
implementations are likely to be gone or, even if they 
are still at the institution, to be unable to remember 
the amount of time they spent on various course 
preparation and delivery activities. The burden of 
providing good data about the expenditure of time 
and other cost elements would be reduced if the 
data were collected each time the course is offered 
using a common digital reporting template. If higher 
education institutions used such a template as a 
regular practice, cost data would likely be not only 
more complete but also more accurate. Having the 
cost information for different versions of a course, 
institutions could combine that information with 
data on the number of successful student outcomes 
achieved by each version of the course to arrive at 
a cost-effectiveness measure. Even educators who 
bristle at institutional efforts to reduce costs can see 
the logic of examining cost effectiveness. 


More direct engagement between researchers 
and higher education institutions appears 
necessary both to improve the quality of 

impact evidence and to enhance organizational 
capacity. Neither technology developers nor higher 
education institutions by themselves are likely to 
generate objective evidence of courseware impacts 
under normal conditions. Collaborations among higher 
education institutions, learning technology providers, 
and researchers will be needed to generate a sound 
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SRI Education 


base of evidence concerning impacts and best 
practices. Moreover, research-practice partnerships 
pairing researchers and higher education institutions 
could help the latter organizations build expectations 
and staff capacity around doing appropriately rigorous 
course redesign impact evaluations. 


At the same time, any funder of social programs 

faces the trade-off between funding more services 

and funding more rigorous evaluations of the impact 

of those services. Finding more cost-effective ways 

to gather rigorous evidence at scale is an ongoing 
challenge. The foundation and SRI have made several 
attempts to reduce evaluation costs by developing 
processes and data templates that could be used by 
either higher education institutions themselves (as in the 
Yarnall, Means, and Wetzel evaluation of the Advanced 
Learning Market Acceleration Project) or the technology 
developer/distributor (as in NGCC) with a limited amount 
of direct researcher support. Neither approach has 
proven fully successful. Ideally, participating colleges 
and universities would embrace the goal of doing 
evidence-based decision-making around course 
redesign/courseware adoption at the same time 
they embrace digital learning. By working closely 
with educational institutions in the early phases of this 
work, evaluators could get better data and co-design 
sustainable evaluation practices with college staff. This 
approach may be tried out as part of the Adaptive 
Courseware for Early Success strategy of the Every 
Learner Everywhere Network. 


Courseware impact evaluations need to broaden 
their to encompass longer-term impacts 
including degree attainment as well as course- 
level outcomes. The foundation has amassed 

course Outcome data for scores of courseware 
implementations over the last eight years. While 

very useful, this database by itself does not answer 
questions about progress against the foundation’s goal 
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of increasing the proportion of low-income and under- 
represented students who attain college degrees or 
other marketable credentials. It is time to start looking 
at student progress over time by asking questions such 
as how students who experienced courseware in their 
gateway courses perform relative to those receiving 
more conventional instruction in future courses in their 
major or pathway. Only by conducting longitudinal 
studies of course performance, credit accumulation, 
retention and completion over time can we understand 


For Courseware Vendors: 


e Following the example of NGCC grantees, 
courseware vendors should work closely 
with early users of their products and plan 
for product refinement cycles that can 
be informed by data from alpha and beta 
product users. 


e Product efficacy claims should be stated 
cautiously and should acknowledge the 


For Higher Education Institutions: 


¢ Colleges and universities should take 
responsibility for generating impact 
estimates--based on comparisons 
controlling for key student characteristics 
and prior achievement--for their course 
redesign efforts, including but not limited to 
those incorporating courseware. 


¢ To become evidence-driven with respect 
to their teaching and learning approaches, 
colleges and universities would do well to 
have a highly placed “champion” responsible 
for promoting course improvement efforts 
and impact evaluations. 
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whether or not investments in digital learning are 
having the intended payoff. 


Implications for actions that courseware vendors, higher 
education institutions, researchers, and funders can take 
to respond to these findings are shown in the boxes 

that follow. Close collaboration between all of these 
sectors will be necessary to produce significant and 
consistent improvements in the learning experiences 
and educational attainments of low-income and under- 
represented minority college students. 


importance of implementation factors and 
aspects of teaching and learning other than 
the courseware. 


¢ Courseware vendors could benefit from 
finding external research partners to help 
them design and execute rigorous studies of 
product implementation and effectiveness. 


Course behaviors and outcomes should be 
measured at multiple time scales (during 
the course, end-of-course, and over multiple 
academic terms) both for students overall 
and for key student subgroups (e.g., low- 
income students, under-represented 
minority students). 


Higher education institutions should 
evaluate their large-scale adoptions of 
next-generation courseware in terms of cost 
effectiveness (i.e., cost per desired student 
outcome) rather than cost reduction per se. 
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For Researchers and Evaluators: 


e Researchers should work closely with 


course designers to define and collect data 
on early indicators of likely course success. 


e Researchers should be advocates for 
stronger course evaluation designs, 
featuring random assignment of students 
to different versions of the course where 
possible or statistical controls for student 
characteristics and prior achievement at a 
minimum. 


For Funders: 


e The field would benefit from an evidence- 
driven deeper dive into the influences of 
instructor quality, instructional practices, 
student and instructor supports, and 
students’ own behaviors and perspectives 
when courseware is introduced. 


¢ Obtaining evidence on the impacts of 
courseware implementation at scale 
may require supporting higher education 
institutions in organizational change 
around the use of evidence as a key 
to improving instruction and student 
outcomes. 
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¢ Courseware impact studies should expand 


their focus to encompass longer-term 
student outcomes, including persistence 
and degree attainment, as well as course 
grades and credits. 


e Researchers should help their higher 


education and courseware vendor 
collaborators understand that expected 
large samples of students using a particular 
learning technology can become small 
when analyses apply appropriate controls 
for student characteristics. 


e More direct engagement between 


researchers and higher education 
institutions appears necessary both to 
improve the quality of impact evidence and 
to enhance organizational capacity. 


e The field needs research that examines 


longer-term impacts of courseware, such 

as student persistence, performance in the 
next course in the same pathway or subject 
area, and credit accumulation over time. 
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Appendix A: Number of Students Using Courseware Products 
by Academic Term 


Table A-1. Number of Students Using Courseware Products by Academic Term 


Spr/Sum —eay2015 0 SP/Sum— cayo01g «= SPF/SUM— ee a9017 Ss TOTAL 


2015 2016 2017 

Acrobatiq : 315 11,992 27,702 8,288 48,297 
Cerego' - 1 OO) 1,029) 1,005 361 - 4,245 
CogBooks : 559 464 959 560 : 2,542 
eee . 1,824 3,902 2,521 : . 8,247 

OpenStax - - 14,316 OrSig2 5, 1se) 653 Sila) 
hae 728 855 1,389 1,423 1,186 : 5,581 

Stanford 7,017 4,594 7,196 5,883 6,888 6,241 37,819 

TOTAL* 7,745 9,382 28,911 34,155 42,456 15,182 137,831 


1 Cerego declined to provide enrollment data for AY 2016-17. 
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Appendix B: Instructor Survey Data 


Table B-1. Aggregate (n=550, 78% Response Rate) 


SRI Education 


Q# Question Text n Mean <1) 

1 Are you currently using PRODUCT in any of your courses? 362 1.00 0.00 

3a “About how many students are enrolled in this section of {Course}?” 
35 or fewer 402 0.92 0.27 
36-99 ali 0.04 0.19 
100-249 3 0.01 0.08 
250-499 4 0.01 0.10 
500 or more i 0.03 0.16 

3b “About how many students are enrolled in this section of {Course}?” 
35 or fewer 68 0.93 0.25 
36-99 3 0.04 0.20 
100-249 1 0.01 0.12 
250-499 1 0.01 0.12 
500 or more ) 

ie “About how many students are enrolled in this section of {Course}?” 
35 or fewer 24 0.92 0.27 
36-99 2 OOS OF2I7 
100-249 0 
250-499 0 
500 or more 0 

3d “About how many students are enrolled in this section of {Course}?” 
35 or fewer 10 0.91 0.30 
36-99 1 0.09 0.30 
100-249 0 
250-499 0 
500 or more 0 

3e “About how many students are enrolled in this section of {Course}?” 
35 or fewer 6 0.86 0.38 
36-99 1 0.14 0.38 
100-249 0 
250-499 0 
500 or more 0 
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Table B-1. Aggregate (n=550, 78% Response Rate) (cont'd.) 


Q# Question Text 


SRI Education 


ral Mean 


) 3) 


Sf “About how many students are enrolled in this section of {Course}?” 


35 or fewer 


100-249 
250-499 


500 or more 


3g “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 


500 or more 


3 to) 


1.00 


ee) |e) |e |) 


50 


0.00 


3h “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 


500 or more 


1.00 


ey ey eS | je) = 


0.00 


3i “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 


500 or more 


(ey | ey | te) | te) |) 


0.00 


3) “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 


500 or more 


1.00 


(fe) | 2) | fey eo = 


0.00 


4 Is this your first time using PRODUCT in COURSE? 
Is this your first time using PRODUCT in COURSE? 
Is this your first time using PRODUCT in COURSE? 
Is this your first time using PRODUCT in COURSE? 
Is this your first time using PRODUCT in COURSE? 
Is this your first time using PRODUCT in COURSE? 
Is this your first time using PRODUCT in COURSE? 
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472 0.54 
58 0.48 
15 0.60 

50 


0.50 
0.50 
0.51 
0.58 
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Table B-1. Aggregate (n=550, 78% Response Rate) (cont'd.) 


Q# Question Text n Mean SD 
5 Have you taught this course before without using {Product}? (1=Yes) 104 0.81 0.40 
5a Did you use another courseware product previously? (1=Yes) 8 0.25 0.46 
Did you teach this course with PRODUCT in a previous term? 180 ORS 0.50 
For how many terms have you used PRODUCT in the past? 
1 term 61 0.34 0.47 
2 terms 44 0.24 0.43 
3 terms 24 0.13 0.34 
4 terms 13 0.07 0.26 
5+ terms 38 0.21 0.44 
10 Have you ever taught courses with online or blended learning components before 472 0.67 0.47 
using PRODUCT? 
11 “Last Fall, were you teaching...” (1=Yes, Full time) 405 0.73 0.44 
12 “Last Fall, was your position...” 
ml Tenure track faculty 209 0.52 0.50 
2 Lecturer or Instructor 95 0.23 0.42 
= Adjunct professor, lecturer, or instructor 101 0.25 0.43 
1é “Did you have a choice to use {Product} for your course this term (spring 2017)?” 450 0.70 0.46 
(1=Yes) 
14 “What motivated you to choose {Product}?” (1=Yes) 
ll Interest in exploring online teaching and learning 314 0.72 0.45 
2 Thought {Product} would help with course management or organization 314 0.50 0.50 
_3 Thought teaching with {Product} would take less of my time 314 0.22 0.42 
_4 Desire for a more flexible teaching schedule 314 OnZ Onss 
mS) Opportunity for professional development 314 0.39 0.49 
6 Received an attractive incentive 314 0.09 0.29 
ays This was the only online learning tool available to me for this course 314 0.09 0.29 
_ To be consistent with other courses/sections 314 0.16 0.37 
9 Desire for a more convenient way to communicate with students 314 0.26 0.44 
_10 Expectation that my students would be more engaged 314 0.87 0.34 
alin Expectation that my students would receive more individualized learning 314 0.84 0.37 
V2 Encouragement from upper levels of your institution (for example, your 314 0.19 0.39 
department or college) to use {Product} 
18 Wanted to same my students the cost of a textbook 18 OMS Or43 
lat Other (please explain) 314 0.31 0.46 
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Table B-1. Aggregate (n=550, 78% Response Rate) (cont'd.) 


Q# Question Text n Mean SD 
13 “To what extent did using {Product} change the way you...” (1=Some Change or 
Major Change) 
_1| Planned this course om Oil 0.50 
_2 Presented the material for this course Suli7 0.53 0.50 
Ss Interacted with students in this course 518 0.49 0.50 
4 Evaluated students in this course Sulla 0.6 0.49 
16 “Did you receive training or professional development this term (spring 2017) onhow 482 0.46 0.50 
to use or integrate {product} into your course to improve student learning?” (1=Yes) 
lif “How many hours of training did you receive this term?” 78 O24 e2al 
19 “How satisfied were you with the training you received this term?” (1=Moderately or 220 0.89 0.32 
Highly Satisfied) 
20 “Was training for {Product} offered this term (spring 2017)?” (1=Yes) 258 0.31 0.46 
21 “Briefly indicate your reasons for not participating in the training for {Product} this 
year.” (1=Yes) 
1 Training did not align with my schedule 78 0.36 0.48 
2 Training offered in inconvenient location or format 78 0.08 0.27 
mc) Training took too much time 76 0.22 0.42 
_4 Training occurred before | was hired 78 0.04 0.19 
6 | took the training previously ial 0.32 0.47 
6 Did not feel the need for training 184 O25 0.43 
ff Other, please describe. 76 0.11 0.31 
22 “The last time you taught your course without {Product}, did you receive any training 52 0.38 0.49 


or professional development on how to design and implement instruction to improve 
student learning?” (1=Yes) 


23 “How many hours of training did you receive for your course on how to design and 19 6.53 8.61 
implement instruction and to improve student learning when not using {Product}?” 


24 “When preparing for your course this spring, how much time did you spend 463 8.37 10.00 
(in hours) learning how to use {Product}, in both formal training and individual 
preparation? Do not include any time spent learning the product before fall 2016. 


25) “When preparing for your course this spring, how much time did you spend (in 460 16.24 15.27 
hours) integrating {Product} into your course (e. g., adapting assignments, adapting 
format of content or lesson delivery, adapting grading, etc.)? Do not include any time 
spent learning the product before fall 2016.” 


26 “How prepared did you feel to use {Product} this term (spring 2017)?” (1=Very or 550 0.69 0.46 
Adequately Prepared) 


2h “Have you received support for using {Product} outside of formal training this term SZ 0.56 0.50 
(spring 2017?” (1=Yes) 
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Table B-1. Aggregate (n=550, 78% Response Rate) (cont'd.) 


Q# Question Text n Mean SD 
28 “From which of these sources have you received support for using {Product} this term? (1=Yes) 
| {Product} support staff/technical assistance 286 0.85 0.36 
2 Next Generation Courseware Challenge project leader on my campus 286 Oe 0.38 
ms) Peers/colleagues also using {Product} 286 0.51 0.50 
_4 Technical (IT) support staff on my campus 286 0.21 0.41 
5 Instructional design staff on my campus 286 0.23 0.42 
& Someone else from my department 286 0.22 0.42 
=f: Other (please specify) 286 0.08 0.27 
29 “How helpful was this support in enabling you to use {Product} in your course this term (spring 2017)?” 
(1=Very Helpful or Helpful) 
1 {Product} support staff/technical assistance 242 0.89 0.32 
2 Next Generation Courseware Challenge project leader on my campus 50 0.92 0.27 
_3 Peers/colleagues also using {Product} 147 0.89 0.31 
_4 Technical (IT) support staff on my campus 58 0.74 0.44 
5 Instructional design staff on my campus 65 0.94 0.24 
6} Someone else from my department 60 0.83 0.38 
i Other (please specify) 13 0.85 0.38 
30 “Which of the following best describes how {Product} is/was used in your course this term (spring 2017)?” 
Substitute for printed text as core course component 174 0.32 0.47 
Substitute for some lecture/class meeting time 2 0.02 ORS 
Supplemental source of course content ilialli74 0.21 0.41 
Skills practice environment 92 0.17 0.37 
Formative assessment system 28 0.05 0.22 
Flipping the classroom (i.e., using the product to cover content in advance so that 48 0.09 0.28 
class time could be used for more active activities) 
Fully online delivery of course 78 0.14 0.35 
31 “What proportion of the content in your course is being delivered using {Product}?” 
0% to 24% 163 0.37 0.48 
25% to 49% 85 0.19 0.40 
50% to 79% 75 0.17 0.38 
80% to 100% 116 0.26 80.44 
o2 “Before the start of your course this term (spring 2017), how many hours would you 61 Weis lel 


estimate you spent planning your instruction and student assignments?” 
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Table B-1. Aggregate (n=550, 78% Response Rate) (cont'd.) 


Q# Question Text n Mean SD 
38a “Think about a typical week teaching your course this term (spring 2017). How many hours did 
you spend on each of the following course-related activities during that week?” 
Presenting course content and leading course instruction hours 61 6.00 5.61 
Interacting with students (including office hours, email correspondence, etc.) 61 8.98 9.42 
Evaluating or providing written feedback on student work 61 5.46 5.82 
Course administration (e.g. entering grades, technology assistance, etc.) 61 4.48 3.84 
Revising your plan for instruction and student assignments 61 Suf2 6.07 
34 “How many hours did you spend planning instruction and student assignments 
before the course started for {Course} with and without using {Product}?” 
1 With {Product} 83 S24 lOnae 
Z| Without using {Product} 82 Wc) LY} 
35 “During a typical week, how many hours did/do you spend performing the following 
tasks for {Course} with and without using {Product}?” 
le With using {Product}: Presenting course content and leading course instruction hours 83 5.16 4.98 
12 With using {Product}: Interacting with students (including office hours, email 83 6.39 6.65 
correspondence, etc.) 
18 With using {Product}: Evaluating or providing written feedback on student work 84 4.27 Chev! 
1_4 With using {Product}: Course administration (e.g. entering grades, technology 83 3.73 3.08 
assistance, etc.) 
1 With using {Product}: Revising your plan for instruction and student assignments 82 3.40 2.65 
2 )| Without using {Product}: Presenting course content and leading course instruction hours 83 5.76 5.20 
2 Without using {Product}: Interacting with students (including office hours, email 83 6.06 Suid) 
correspondence, etc.) 
2n3 Without using {Product}: Evaluating or providing written feedback on student work 83 4.89 4.53 
2 4 Without using {Product}: Course administration (e.g. entering grades, technology 83 3.83 8.22 
assistance, etc.) 
2ao) Without using {Product}: Revising your plan for instruction and student 82 3.66 Oral 
assignments 
36 “Overall, how would you describe the impact of using {Product} on the time you 144 0.38 0.49 
spend on your course?” (1= Major or Minor Time Saver) 
37 How important is the ability to print content to use PRODUCT? 68 0.56 0.50 
38 Who pays for printing? 
College pays for printing We 0.45 0.50 
Students pays for printing 16 0.42 0.50 
Other pays for printing 5) 0.13 0.34 
39 Approximately how many pages per student are printed over the entire course? 
0-50 pages lS 0.63 0.49 
51-100 pages 2 0.08 0.28 
101-150 pages 3 0.13 0.34 
151-200 pages 1 0.04 0.20 
200+ pages 3 0.13 0.34 
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Table B-1. Aggregate (n=550, 78% Response Rate) (cont'd.) 
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Q# Question Text n Mean SD 
40 “When your course is taught at your institution without {Product}, are textbooks or 143 0.85 0.36 
other materials generally required or recommended for students?” (1=Yes) 
4 “What is the approximate cost of text and other materials for your course when 
teaching it without {Product}?” (1=Less than $25, 7=More than $275) 
Less than $25 18 0.15 0.36 
$26-$75 ia 0.09 0.29 
$76-$125 30 0.25 0.44 
$126-$175 ile 0.14 0.35 
$176-$225 29 O25 9) 0:48 
$226-$275 0.04 0.20 
More than $275 0.03 0.16 
Don’t know 0.04 0.20 
42 “Is your course staffed differently when using {Product}?” 
al Yes, more staff 23 0.17 0.37 
= Yes, less staff il 0.08 0.27 
= No, same staff 97 0.70 0.46 
= Don’t Know i 0.05 0.22 
43 “How frequently do you use each of the following instructor-oriented features of 
{Product}?” (1=Used All or Some of the Time) 
1 Used online gradebook to keep track of individual students’ performance 528 0.67 0.47 
2 Used dashboards to track progress of the class as a whole 535 0.49 0.50 
8 Used dashboard to check individual students’ progress in understanding concepts 5382. 0.52 0.50 
and content 
4 Used dashboard to check when individual students last logged in to {Product} 529 0.33 0.47 
_& Used {Product} student performance data to refer individual students to specific 533 0.32 0.47 
content or problem activities 
_6 Adjusted the structure or sequencing of course content in {Product} for some 522 0.17 0.37 
students 
_ff Used {Product} to communicate with students 520 0.22 0.42 
8} Used {Product} to make class materials available to students 520 0.47 0.50 
_9 Identifying common misconceptions so that | could address them during class time 530 0.34 0.47 
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Table B-1. Aggregate (n=550, 78% Response Rate) (cont'd.) 


Q# Question Text n Mean SD 
44 “How much do you agree or disagree with the following statements? Using {Product} 
helps students...” (1=Strongly Agree or Agree) 
i Better understand and remember course content 546 0.66 0.47 
= Better analyze and apply course content 546 0.62 0.49 
Bs Complete more complex assignments that demonstrate their understanding of 543 0.48 0.50 
course content 
_4 Stay more engaged in the course 546 0.64 0.48 
6) Reduce costs associated with the course (e.g., purchasing a textbook) 543 0.42 0.49 
mG) Interact more with me around course content 436 0.78 0.42 
me Interact more with other students around course content 545 0.34 0.47 
i Other (please explain) 68 0.46 0.50 
45 “How much do you agree or disagree with the following statements regarding 
{Product}? Using {Product} in this course helped me...” (1=Strongly Agree or Agree) 
ml Present content more effectively 545 0.52 0.50 
=e Personalize learning in response to different students’ needs 542 0.48 0.50 
as} Reduce time spent assessing student learning 547 0.44 0.49 
_4 Reduce the time spent on mechanical teaching tasks such as recording grades 545 0.35 0.48 
8) Reduce time spent in class 539 0.21 0.41 
m6 Better monitor individual student progress 548 OSG 0.50 
aff Keep better track of whole-class progress 546 0.53 0.50 
3} Increase student engagement with the content of my course 545 0.63 0.48 
9) Use class time for more interactive activities 541 0.47 0.50 
10 Reduce time spent preparing the course at the start of the term 433 0.24 0.43 
eeliil Support my students in deeper learning 546 0.63 0.48 
mil Other 98 0.58 0.50 
46 “Did you experience any usability and technical problems when using {Product} in 516 0.56 0.50 
your course this term (spring 2017)?” (1=Yes) 
47 “Which of these usability and technical problems did you experience when using 
{Product} in your course this term?” (1=Yes) 
el Problems with logging or signing in (either you or your students) 289 0.55 0.50 
Ee: Incompatibilities with existing computer hardware or software (e.g., browser 289 0.47 0.50 
incompatibilities) 
6 Incompatibilities with our campus systems, such as the Learning Management or 289 0.27 0.45 
Student Information System 
_4 Incompatibilities with other online resources 289 0.12 0.38 
_&) Lack of technical or help desk support needed from my campus 289 0.19 0.40 
6 Lack of technical or help desk support needed from {Grantee} 289 0.20 0.40 
all Confusing design of dashboard or system interface 289 0.44 0.50 
8 Other 284 0.42 0.49 
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Table B-1. Aggregate (n=550, 78% Response Rate) (cont'd.) 


Q# Question Text n Mean SD 
48 “Which, if any, of these problems did you experience when using {Product} in your 
course this term (spring 2017)?” (1=Yes) 
1 Lack of alignment with course goals 507 0.19 0.39 
ee Time required to learn new online system 507 0.27 0.44 
8} Student resistance to using system 507 0.42 0.49 
4 Students did not use {Product} the way it was intended (i.e., attempting to succeed 507 0.39 0.49 
by exploiting properties of the system rather than learning the material) 
6) Did not work well within course schedule or allotted class time 507 0.09 0.29 
S Other, please explain 404 0.16 0.36 
i None of these 389 0.06 0.25 
49 “Overall, how satisfied have you been when using {Product} in your course this term 548 0.88 0.54 
(spring 2017)?” (1=Highly or Moderately Satisfied) 
Sil “Do you plan to use {Product} in your course again?” 
Yes, in much the same way | used it this term 166 0.30 0.46 
Yes, but in a more limited way 227 0.42 0.49 
Yes, and to a greater extent than | used it this term 57 0.10 0.31 
No DS 0.10 0.30 
Not sure 40 0.07 0.26 
52 “How likely are you to recommend {Product} to a friend or colleague teaching the 544 6.52 2.82 


same course you do?” (0 to 10) 


Table B-2. Acrobatiq, Product (n=32, 85% response rate) 


Q# Question Text n LY (Fe 1g =) B) 
1 Are you currently using PRODUCT in any of your courses? 14 1.00 0.00 

Ba “About how many students are enrolled in this section of {Course}?” 
35 or fewer 16 ORS CORON 
36-99 1 0.038 0.18 
100-249 O 0.00 0.00 
250-499 2 OLO2OrZ5 
500 or more W 0.37 0.49 

8b “About how many students are enrolled in this section of {Course}?” 
35 or fewer 0 0.00 
36-99 O 0.00 
100-249 O 0.00 
250-499 1 D5 5 
500 or more 0 0.00 
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Table B-2. Acrobatiq, Product (n=32, 85% response rate) (cont'd..) 


Q# Question Text 


3c “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 


500 or more 


SRI Education 


n Mean SD 


1.00 
0.00 
0.00 
0.00 
0.00 


oo © eS ho 


8d “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 


500 or more 


(ere) |e) |e) = 


3e “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 
500 or more 


eo oe |e |e 


of “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 


500 or more 


Se) fe) eo 


3g “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 
500 or more 


eo ee) | & 


3h “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 


500 or more 


(eo Ve) | @ |e) |=) 
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Table B-2. Acrobatiq, Product (n=32, 85% response rate) (cont'd..) 


Q# Question Text 


3i “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 
500 or more 


QO oe) eo | ce | & 


3j “About how many students are enrolled in this section of {Course}?” 
35 or fewer 
36-99 
100-249 
250-499 
500 or more 
4 Is this your first academic term using PRODUCT in COURSE? 
Is this your first academic term using PRODUCT in COURSE? 
Is this your first academic term using PRODUCT in COURSE? 3 0.00 0.00 
Is this your first academic term using PRODUCT in COURSE? 1 0.00 
Is this your first academic term using PRODUCT in COURSE? 1 0.00 
0 
0 


S| oa | @ | & | © 


es 
™ 


0.36 0.50 


Is this your first academic term using PRODUCT in COURSE? 
Is this your first academic term using PRODUCT in COURSE? 


i Have you taught this course before without using {Product}? (1=Yes) 18 0.28 0.46 

5a Did you use another courseware product previously? (1=Yes) 0 0.00 0.00 
Did you teach this course with PRODUCT in a previous term? 10 0.71 0.47 
For how many terms have you used PRODUCT in the past? 
1 term 1 0.10 0.32 
2 terms 1 0.10 0.32 
3 terms 1 0.10 0.32 
4 terms 3 0.30 0.48 
5+ terms 4 0.40 0.52 

10 Have you ever taught courses with online or blended learning components before 14 OF 0.51 
using PRODUCT? 

lial “Last Fall, were you teaching...” (1=Yes, Full time) 382 als) 0.42 

12 “Last Fall, was your position...” 

al Tenure track faculty i 0.22 0.42 

_2 Lecturer or Instructor 1@ O59 0.50 

6 Adjunct professor, lecturer, or instructor 6 0.19 0.40 
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Table B-2. Acrobatiq, Product (n=32, 85% response rate) (cont'd.) 


Q# Question Text n Mean SD 
1 “Did you have a choice to use {Product} for your course this term (spring 2017)?” 32 0.22 0.42 
(1=Yes) 
14 “What motivated you to choose {Product}?” (1=Yes) 
ml Interest in exploring online teaching and learning 7 Oa 0.49 
2 Thought {Product} would help with course management or organization a 0.71 0.49 
8) Thought teaching with {Product} would take less of my time i 0.43 0.53 
al Desire for a more flexible teaching schedule iG 0.29 0.49 
ZS Opportunity for professional development il 0.43 0.53 
_6 Received an attractive incentive a 0.00 0.00 
_if This was the only online learning tool available to me for this course i 0.14 0.38 
8 To be consistent with other courses/sections i 0.14 0.38 
9 Desire for a more convenient way to communicate with students 7 0.43 O53 
a1@ Expectation that my students would be more engaged i 1.00 0.00 
1 Expectation that my students would receive more individualized learning i 1.00 0.00 
IZ Encouragement from upper levels of your institution (for example, your i 0.14 0.38 
department or college) to use {Product} 
18 Wanted to same my students the cost of a textbook i 
_14 Other (please explain) i 0.29 0.49 
18 “To what extent did using {Product} change the way you...” (1=Some Change or 
Major Change) 
1 Planned this course OZ 0.56 0.50 
2 Presented the material for this course 32 0.59 0.50 
mS Interacted with students in this course Be Ol538 @zSil 
_4 Evaluated students in this course 32. 0.41 0.50 
16 “Did you receive training or professional development this term (spring 2017) on 32 0.44 0.50 
how to use or integrate {product} into your course to improve student learning?” 
(1=Yes) 
lif “How many hours of training did you receive this term?” illal 5.91 as} 
19 “How satisfied were you with the training you received this term?” (1=Moderately or 14 0.79 0.43 
Highly Satisfied) 
20 “Was training for {Product} offered this term (spring 2017)?” (1=Yes) 18 0.33 0.49 
ZA “Briefly indicate your reasons for not participating in the training for {Product} this 
year.” (1=Yes) 
zal Training did not align with my schedule 6 0.33 0.52 
2 Training offered in inconvenient location or format 6 0.00 0.00 
8) Training took too much time 6 0.33 0.52 
_4 Training occurred before | was hired 6 0.00 0.00 
8 | took the training previously 6 Ona 0.41 
6 Did not feel the need for training 6 0.67 0.52 
ff Other, please describe. 6 0.00 0.00 
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Q# Question Text n iY (eT es) B) 
22 “The last time you taught your course without {Product}, did you receive any 5 0.40 ORS5 
training or professional development on how to design and implement instruction to 
improve student learning?” (1=Yes) 
23 “How many hours of training did you receive for your course on how to design and 2 650) 6.36 
implement instruction and to improve student learning when not using {Product}?” 
24 “When preparing for your course this spring, how much time did you spend 26 8.31 12.95 
(in hours) learning how to use {Product}, in both formal training and individual 
preparation? Do not include any time spent learning the product before fall 2016. 
2S) “When preparing for your course this spring, how much time did you spend (in ZU 9.59 13.80 
hours) integrating {Product} into your course (e. g., adapting assignments, adapting 
format of content or lesson delivery, adapting grading, etc.)? Do not include any 
time spent learning the product before fall 2016.” 
26 “How prepared did you feel to use {Product} this term (spring 2017)?” (1=Very or 32 0.84 0.37 
Adequately Prepared) 
ile “Have you received support for using {Product} outside of formal training this term 32 0.59 0.50 
(spring 2017?” (1=Yes) 
28 “From which of these sources have you received support for using {Product} this term? (1=Yes) 
el {Product} support staff/technical assistance 19 0.53 0.51 
m2 Next Generation Courseware Challenge project leader on my campus 19 0.00 0.00 
é Peers/colleagues also using {Product} 19 0.68 0.48 
_4 Technical (IT) support staff on my campus 19 0.26 0.45 
§ Instructional design staff on my campus 19 0.42 0.51 
mo Someone else from my department 19 0.21 0.42 
aif Other (please specify) 19 0.11 0.32 
29 “How helpful was this support in enabling you to use {Product} in your course this term (spring 2017)?” 
(1=Very Helpful or Helpful) 
mil {Product} support staff/technical assistance 10 0.70 0.48 
_p Next Generation Courseware Challenge project leader on my campus O 
ae Peers/colleagues also using {Product} 13 1.00 0.00 
_4 Technical (IT) support staff on my campus 5 0.60 ORS5 
6 Instructional design staff on my campus 8 1.00 0.00 
_6 Someone else from my department 4 1.00 0.00 
_if Other (please specify) 2 1.00 0.00 
30 “Which of the following best describes how {Product} is/was used in your course this term (spring 2017)?” 
Substitute for printed text as core course component 18 0.56 0.50 
Substitute for some lecture/class meeting time O 0.00 0.00 
Supplemental source of course content O 0.00 0.00 
Skills practice environment 1 0.03 0.18 
Formative assessment system O 0.00 0.00 
Flipping the classroom (i.e., using the product to cover content in advance so 4 0.13 0.34 
that class time could be used for more active activities) 
Fully online delivery of course 9 0.28 0.46 
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Table B-2. Acrobatiq, Product (n=32, 85% response rate) (cont'd.) 


Q# Question Text n Mean SD 
31 “What proportion of the content in your course is being delivered using {Product}?” 
0% to 24% 0 0.00 0.00 
25% to 49% 7 O22) 0.42 
50% to 79% 5) 0.16 0.37 
80% to 100% 20 0.63 0.49 
Ce “Before the start of your course this term (spring 2017), how many hours would you is) 13.08 14.16 
estimate you spent planning your instruction and student assignments?” 
38a “Think about a typical week teaching your course this term (spring 2017). How 
many hours did you spend on each of the following course-related activities during 
that week?” 
Presenting course content and leading course instruction hours 13 6.08 10.78 
Interacting with students (including office hours, email correspondence, etc.) is ie O2 elloree, 
Evaluating or providing written feedback on student work 13 OOM 23 
Course administration (e.g. entering grades, technology assistance, etc.) 13 2.54 1.98 
Revising your plan for instruction and student assignments 13 5.46 11.15 
34 “How many hours did you spend planning instruction and student assignments 
before the course started for {Course} with and without using {Product}?” 
lea With {Product} 5 5.80 5.02 
ml Without using {Product} 5 12.60 16.88 
35 “During a typical week, how many hours did/do you spend performing the following 
tasks for {Course} with and without using {Product}?” 
qu With using {Product}: Presenting course content and leading course instruction 5) 3.80 0.84 
hours 
12 With using {Product}: Interacting with students (including office hours, email 5 15.60 10.53 
correspondence, etc.) 
sli With using {Product}: Evaluating or providing written feedback on student work 5 3.00 3.39 
We With using {Product}: Course administration (e.g. entering grades, technology 5 2.60 1.14 
assistance, etc.) 
slr) With using {Product}: Revising your plan for instruction and student assignments © 2.60 1.34 
2 Without using {Product}: Presenting course content and leading course 5 5.20 Zot 
instruction hours 
22 Without using {Product}: Interacting with students (including office hours, email o) 13.60 11.99 
correspondence, etc.) 
2.3 Without using {Product}: Evaluating or providing written feedback on student 5) 3.80 3.56 
work 
2 4 Without using {Product}: Course administration (e.g. entering grades, technology 6) 3.40 1.82 
assistance, etc.) 
2S Without using {Product}: Revising your plan for instruction and student 5) 3.80 7S: 
assignments 
36 “Overall, how would you describe the impact of using {Product} on the time you 18 ORI 0.38 
spend on your course?” (1= Major or Minor Time Saver) 
37 How important is the ability to print content to use PRODUCT? O 
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Table B-2. Acrobatiq, Product (n=32, 85% response rate) (cont'd.) 
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Q# Question Text n Mean SD 
38 Who pays for printing? 
College pays for printing 
Students pays for printing 
Other pays for printing 0 
39 Approximately how many pages per student are printed over the entire course? 
0-50 pages O 
51-100 pages 0 
101-150 pages 0 
151-200 pages 0 
200+ pages 0) 
40 “When your course is taught at your institution without {Product}, are textbooks or 18 0.44 OES 
other materials generally required or recommended for students?” (1=Yes) 
41 “What is the approximate cost of text and other materials for your course when 
teaching it without {Product}?” (1=Less than $25, 7=More than $275) 
Less than $25 3 0.38 0.52 
$26-$75 1 Once Oca 
$76-$125 0 0.00 0.00 
$126-$175 0 0.00 0.00 
$176-$225 0 0.00 0.00 
$226-$275 0 0.00 0.00 
More than $275 0 0.00 0.00 
Don’t know 4 0.50 0.53 
42 “Is your course staffed differently when using {Product}?” 
ai Yes, more staff 2 lial 32 
aS Yes, less staff 0.00 0.00 
= No, same staff 13 Ge 46 
es Don’t Know Ont 0.38 
43 “How frequently do you use each of the following instructor-oriented features of 
{Product}?” (1=Used All or Some of the Time) 
me Used online gradebook to keep track of individual students’ performance 32 0.44 0.50 
Used dashboards to track progress of the class as a whole 32 0.34 0.48 
=e Used dashboard to check individual students’ progress in understanding concepts 31 0.42 0.50 
and content 
_4 Used dashboard to check when individual students last logged in to {Product} 32 0.44 OF5O) 
a Used {Product} student performance data to refer individual students to specific 32 0.31 0.47 
content or problem activities 
a6 Adjusted the structure or sequencing of course content in {Product} for some students Sil ORS 0.34 
ff Used {Product} to communicate with students 31 0.13 0.34 
8} Used {Product} to make class materials available to students 32 0.44 0.50 
_9 Identifying common misconceptions so that | could address them during class time 382 0.38 0.49 
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Table B-2. Acrobatiq, Product (n=32, 85% response rate) (cont'd.) 


Q# Question Text n Mean SD 
44 “How much do you agree or disagree with the following statements? Using 
{Product} helps students...” (1=Strongly Agree or Agree) 
_l Better understand and remember course content 31 0.61 0.50 
es Better analyze and apply course content 31 OFS 0.51 
6 Complete more complex assignments that demonstrate their understanding of Bi 0.23 0.43 
course content 
_4 Stay more engaged in the course 31 Or52 0.51 
6 Reduce costs associated with the course (e.g., purchasing a textbook) Si 0.35 0.49 
mo Interact more with me around course content 31 0.61 0.50 
mi Interact more with other students around course content 31 0.29 0.46 
ZS Other (please explain) 5 0.60 OF55 
45 “How much do you agree or disagree with the following statements regarding 
{Product}? Using {Product} in this course helped me...” (1=Strongly Agree or Agree) 
I Present content more effectively Bi O55 @ESil 
2 Personalize learning in response to different students’ needs 30 0.33 0.48 
3) Reduce time spent assessing student learning 31 0.32 0.48 
_4 Reduce the time spent on mechanical teaching tasks such as recording grades 30 0.27 0.45 
zo Reduce time spent in class 29 0.21 0.41 
mG Better monitor individual student progress 31 0.35 0.49 
ff Keep better track of whole-class progress 30 0.37 0.49 
_8 Increase student engagement with the content of my course 30 0.53 0.51 
9) Use class time for more interactive activities 30 0.30 0.47 
_10 Reduce time spent preparing the course at the start of the term 30 Os 0.43 
mal Support my students in deeper learning oil 0.35 0.49 
mal Other 3 0.33 0.58 
46 “Did you experience any usability and technical problems when using {Product} in 32 0.63 0.49 
your course this term (spring 2017)?” (1=Yes) 
47 “Which of these usability and technical problems did you experience when using 
{Product} in your course this term?” (1=Yes) 
| Problems with logging or signing in (either you or your students) 20 0.45 0.51 
2 Incompatibilities with existing computer hardware or software (e.g., browser 20 0.25 0.44 
incompatibilities) 
_3 Incompatibilities with our campus systems, such as the Learning Management or 20 0.35 0.49 
Student Information System 
_4 Incompatibilities with other online resources 20 ORNS 0.37 
_& Lack of technical or help desk support needed from my campus 20 0.10 0.31 
6 Lack of technical or help desk support needed from {Grantee} 20 0.40 0.50 
if Confusing design of dashboard or system interface 20 0.35 0.49 
_& Other 20 0.30 0.47 
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Table B-2. Acrobatiq, Product (n=32, 85% response rate) (cont'd.) 


Q# Question Text n Mean SD 
48 “Which, if any, of these problems did you experience when using {Product} in your 
course this term (spring 2017)?” (1=Yes) 
=| Lack of alignment with course goals 31 0.16 0.37 
2 Time required to learn new online system 31 0.13 0.34 
8) Student resistance to using system 31 0.29 0.46 
_4 Students did not use {Product} the way it was intended (i.e., attempting to 31 O85 0.49 
succeed by exploiting properties of the system rather than learning the material) 
e Did not work well within course schedule or allotted class time Sil 0.10 0.30 
me Other, please explain il 0.11 FSi 
mi None of these 31 0.33 0.48 
49 “Overall, how satisfied have you been when using {Product} in your course this term 82 0.69 0.47 
(spring 2017)?” (1=Highly or Moderately Satisfied) 
oil “Do you plan to use {Product} in your course again?” 
Yes, in much the same way | used it this term 16 0.50 0.51 
Yes, but in a more limited way 9 0.28 0.46 
Yes, and to a greater extent than | used it this term 1 0.03 0.18 
No 3 0.09 0.30 
Not sure 3 0.09 0.30 
52 “How likely are you to recommend {Product} to a friend or colleague teaching the 31 4.48 3.68 


same course you do?” (0 to 10) 


Table B-3. Cerego, Product (n=26, 78% response rate) 


Q# Question Text n Mean SD 
1 Are you currently using PRODUCT in any of your courses? lS 1.00 0.00 
8a “About how many students are enrolled in this section of {Course}?” 
35 or fewer 15 1 
36-99 
100-249 
250-499 
500 or more 
3b “About how many students are enrolled in this section of {Course}?” 
35 or fewer 
36-99 
100-249 
250-499 
500 or more 


=k, 
i 


je | @ | eo |e 
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Table B-3. Cerego, Product (n=26, 78% response rate), (Cont'd.) 


Q# Question Text 


oC “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 


500 or more 


SRI Education 


(=<) es) B) 


ey | 2) oe oe) | © 


3d “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 


500 or more 


oe |e | oe | eo) & 


3e “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 


500 or more 


je | © | @ | CG} So 


Si “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 


500 or more 


3g “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 
500 or more 


je |e) | ~) | ey | = 


ee) ea) ie) | ©) 


Sh “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 
500 or more 


@) |e |e! |e) | ©) 
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Table B-3. Cerego, Product (n=26, 78% response rate) (cont'd.) 


Q# Question Text n Mean SD 
3i “About how many students are enrolled in this section of {Course}?” 
35 or fewer 0 
36-99 0 
100-249 0 
250-499 0 
500 or more 0 
3) “About how many students are enrolled in this section of {Course}?” 
35 or fewer 0 
36-99 0 
100-249 0 
250-499 0 
500 or more ) 
4 Is this your first time using PRODUCT in COURSE? 
Is this your first time using PRODUCT in COURSE? 26 0.50 0.51 
Is this your first time using PRODUCT in COURSE? 6 0.50 0.55 
Is this your first time using PRODUCT in COURSE? 0 
Is this your first time using PRODUCT in COURSE? 0 
Is this your first time using PRODUCT in COURSE? 0 
Is this your first time using PRODUCT in COURSE? 0 
& Have you taught this course before without using {Product}? (1=Yes) 2 1.00 0.00 
5a Did you use another courseware product previously? (1=Yes) 0 0.00 0.00 
Did you teach this course with PRODUCT in a previous term? 15 0.47 OFZ 
For how many terms have you used PRODUCT in the past? 
1 term 2 0.29 0.49 
2 terms 2 0.29 0.49 
3 terms 0 0.00 0.00 
4 terms 0 0.00 0.00 
5+ terms 3 0.43 0.53 
10 Have you ever taught courses with online or blended learning components before 26 OREG 0.43 
using PRODUCT? 
1 “Last Fall, were you teaching...” (1=Yes, Full time) 1s .60 OFSrl 
12 “Last Fall, was your position...” 
mal Tenure track faculty 5 0.33 0.49 
ae Lecturer or Instructor 5 0.33 0.49 
Adjunct professor, lecturer, or instructor 5 0.33 0.49 
1s es have a choice to use {Product} for your course this term (spring 2017)?” 18 ON8s 0.38 
=Yes 
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Table B-3. Cerego, Product (n=26, 78% response rate) (cont'd.) 


Q# Question Text n Mean SD 

14 “What motivated you to choose {Product}?” (1=Yes) 

= Interest in exploring online teaching and learning 18 0.67 0.49 
2 Thought {Product} would help with course management or organization 15 0.20 0.41 
_3 Thought teaching with {Product} would take less of my time lS 0.07 0.26 
4 Desire for a more flexible teaching schedule 18 0.00 0.00 
_& Opportunity for professional development iS 0.27 0.46 
6 Received an attractive incentive 15 0.40 0.51 
_if This was the only online learning tool available to me for this course LS 0.00 0.00 
as) To be consistent with other courses/sections 15 0.07 0.26 
9 Desire for a more convenient way to communicate with students 15) 0.07 0.26 
10 Expectation that my students would be more engaged iS 0.87 0.35 
pala Expectation that my students would receive more individualized learning 18 0.93 0.26 
12 Encouragement from upper levels of your institution (for example, your lS 0.00 0.00 


department or college) to use {Product} 
ils) Wanted to same my students the cost of a textbook 


_14 Other (please explain) 15 0.20 0.41 
i) “To what extent did using {Product} change the way you...” (1=Some Change or 
Major Change) 
ml Planned this course 26 OF35 0.49 
2 Presented the material for this course 26 0.23 0.43 
8) Interacted with students in this course 26 0.42 0.50 
_4 Evaluated students in this course 26 0.54 OrSal 
16 “Did you receive training or professional development this term (spring 2017) on 18 0.56 OFS 
how to use or integrate {product} into your course to improve student learning?” 
(1=Yes) 
le “How many hours of training did you receive this term?” 3 OW 0.58 
19 “How satisfied were you with the training you received this term?” (1=Moderately or 10 0.70 0.48 
Highly Satisfied) 
20 “Was training for {Product} offered this term (spring 2017)?” (1=Yes) 8 0.38 Ona2 
21 “Briefly indicate your reasons for not participating in the training for {Product} this 
year.” (1=Yes) 
=1 Training did not align with my schedule 3 0.33 0.58 
ae Training offered in inconvenient location or format S) 0.00 0.00 
_3 Training took too much time 3 0.33 0.58 
_4 Training occurred before | was hired S) 0.00 0.00 
mS) | took the training previously 1 0.00 
6 Did not feel the need for training 12 On 0.39 
aff Other, please describe. 3 0.00 0.00 
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Table B-3. Cerego, Product (n=26, 78% response rate) (cont'd.) 


SRI Education 


Q# Question Text Mean SD 
22 “The last time you taught your course without {Product}, did you receive any 2 0.00 
training or professional development on how to design and implement instruction to 
improve student learning?” (1=Yes) 
23 “How many hours of training did you receive for your course on how to design and O 
implement instruction and to improve student learning when not using {Product}?” 
24 “When preparing for your course this spring, how much time did you spend 24 Ws Ae 
(in hours) learning how to use {Product}, in both formal training and individual 
preparation? Do not include any time spent learning the product before fall 2016. 
2S) “When preparing for your course this spring, how much time did you spend (in 24 i SO 4225) 
hours) integrating {Product} into your course (e. g., adapting assignments, adapting 
format of content or lesson delivery, adapting grading, etc.)? Do not include any 
time spent learning the product before fall 2016.” 
26 “How prepared did you feel to use {Product} this term (spring 2017)?” (1=Very or 26 0.54 0.51 
Adequately Prepared) 
Zi “Have you received support for using {Product} outside of formal training this term 26 0.81 0.40 
(spring 2017?” (1=Yes) 
28 “From which of these sources have you received support for using {Product} this term? (1=Yes) 
ea {Product} support staff/technical assistance 21 0.95 0.22 
2 Next Generation Courseware Challenge project leader on my campus 21 0.14 0.36 
8} Peers/colleagues also using {Product} 21 0.29 0.46 
_4 Technical (IT) support staff on my campus 21 0.05 0.22 
_§ Instructional design staff on my campus 21 0.19 0.40 
_6 Someone else from my department 21 0.10 0.30 
mal Other (please specify) 21 0.00 0.00 
29 “How helpful was this support in enabling you to use {Product} in your course this term (spring 2017)”?” 
(1=Very Helpful or Helpful) 
si {Product} support staff/technical assistance 20 0.85 Ore 
2 Next Generation Courseware Challenge project leader on my campus 1.00 0.00 
3 Peers/colleagues also using {Product} 6 1.00 0.00 
_4 Technical (IT) support staff on my campus 1 1.00 
_& Instructional design staff on my campus 4 1.00 0.00 
_6 Someone else from my department 2) 1.00 0.00 
me Other (please specify) O 
30 “Which of the following best describes how {Product} is/was used in your course this term (spring 2017)?” 
Substitute for printed text as core course component 1 0.04 0.20 
Substitute for some lecture/class meeting time 0.00 0.00 
Supplemental source of course content 0.35 0.49 
Skills practice environment 13 0.50 0.51 
Formative assessment system Onl 0.33 
Flipping the classroom (i.e., using the product to cover content in advance so O 0.00 0.00 
that class time could be used for more active activities) 
Fully online delivery of course O 0.00 0.00 
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Table B-3. Cerego, Product (n=26, 78% response rate) (cont'd.) 


Q# Question Text n Mean SD 
Sil “What proportion of the content in your course is being delivered using {Product}?” 
0% to 24% 14 0.93 0.26 
25% to 49% 0 0.00 0.00 
50% to 79% 0 0.00 0.00 
80% to 100% 1 0.07 0.26 
oo “Before the start of your course this term (spring 2017), how many hours would you 5 8.00 4.06 
estimate you spent planning your instruction and student assignments?” 
38a “Think about a typical week teaching your course this term (spring 2017). How 
many hours did you spend on each of the following course-related activities during 
that week?” 
Presenting course content and leading course instruction hours 6 S250 1.64 
Interacting with students (including office hours, email correspondence, etc.) 6 6.67 5.39 
Evaluating or providing written feedback on student work 6 3.67 0.82 
Course administration (e.g. entering grades, technology assistance, etc.) 6 2.33 1.03 
Revising your plan for instruction and student assignments 6 6.33 5.24 
34 “How many hours did you spend planning instruction and student assignments 
before the course started for {Course} with and without using {Product}?” 
Vl With {Product} 2 21.00 0.00 
21 Without using {Product} 2 GOO Same O17 
85 “During a typical week, how many hours did/do you spend performing the following 
tasks for {Course} with and without using {Product}?” 
1 With using {Product}: Presenting course content and leading course instruction 2 4.00 0.00 
hours 
12 With using {Product}: Interacting with students (including office hours, email 2 6.00 4.24 
correspondence, etc.) 
1_3 With using {Product}: Evaluating or providing written feedback on student work 2 4.00 1.41 
1_4 With using {Product}: Course administration (e.g. entering grades, technology 2 5.50 OR val 
assistance, etc.) 
1S With using {Product}: Revising your plan for instruction and student assignments 2 3.00 1.41 
2a Without using {Product}: Presenting course content and leading course 2 4.00 0.00 
instruction hours 
2a2 Without using {Product}: Interacting with students (including office hours, email 2 6.00 4.24 
correspondence, etc.) 
2.3 Without using {Product}: Evaluating or providing written feedback on student 2, 8.50 Baie 
work 
2 4 Without using {Product}: Course administration (e.g. entering grades, technology 2 5.00 0.00 
assistance, etc.) 
2.5 Without using {Product}: Revising your plan for instruction and student 2 3.00 1.41 
assignments 
36 “Overall, how would you describe the impact of using {Product} on the time you 8 0.25 0.46 
spend on your course?” (1= Major or Minor Time Saver) 
ai How important is the ability to print content to use PRODUCT? 8 0.25 0.46 


Next Generation Courseware Challenge Evaluation 102 


Table B-3. Cerego, Product (n=26, 78% response rate) (cont'd.) 
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Q# Question Text Mean SD 
38 Who pays for printing? 
College pays for printing 1.00 0.00 
Students pays for printing 0.00 0.00 
Other pays for printing 0.00 0.00 
39 Approximately how many pages per student are printed over the entire course? 
0-50 pages 1 1.00 
51-100 pages O 0.00 
101-150 pages O 0.00 
151-200 pages O 0.00 
200+ pages O 0.00 
40 “When your course is taught at your institution without {Product}, are textbooks or 8 1.00 0.00 
other materials generally required or recommended for students?” (1=Yes) 
41 “What is the approximate cost of text and other materials for your course when 
teaching it without {Product}?” (1=Less than $25, 7=More than $275) 
Less than $25 0.00 0.00 
$26-$75 0 0.00 0.00 
$76-$125 1 0.13 0.35 
$126-$175 2 0.25 0.46 
$176-$225 5 Oe O52 
$226-$275 0 0.00 0.00 
More than $275 0 0.00 0.00 
Don’t know 0 0.00 0.00 
42 “Is your course staffed differently when using {Product}?” 
1 Yes, more staff 1 0.13 0.35 
m2 Yes, less staff 0 0.00 0.00 
zs No, same staff it 0.88 0.35 
_4 Don’t Know 0 0.00 0.00 
43 “How frequently do you use each of the following instructor-oriented features of 
{Product}?” (1=Used All or Some of the Time) 
al Used online gradebook to keep track of individual students’ performance 25 0.32 0.48 
ne Used dashboards to track progress of the class as a whole 25 0.40 0.50 
_3 Used dashboard to check individual students’ progress in understanding concepts 24 0.58 0.50 
and content 
Me Used dashboard to check when individual students last logged in to {Product} 23 0.35 0.49 
mS) Used {Product} student performance data to refer individual students to specific 25 0.16 ONS 
content or problem activities 
6 Adjusted the structure or sequencing of course content in {Product} for some students 25 0.08 0.28 
mi Used {Product} to communicate with students 24 0.08 0.28 
_8 Used {Product} to make class materials available to students 25 0.16 ONG 
9 Identifying common misconceptions so that | could address them during class time 24 0.13 0.34 
Next Generation Courseware Challenge Evaluation 103 


SRI Education 


Table B-3. Cerego, Product (n=26, 78% response rate) (cont'd.) 


Q# Question Text n Mean SD 
44 “How much do you agree or disagree with the following statements? Using 
{Product} helps students...” (1=Strongly Agree or Agree) 
1 Better understand and remember course content 25 0.80 0.41 
2 Better analyze and apply course content 25 0.60 0.50 
i. Complete more complex assignments that demonstrate their understanding of 25 0.44 OFS 
course content 
_4 Stay more engaged in the course ZS) Onn2 0.46 
mS) Reduce costs associated with the course (e.g., purchasing a textbook) 25 0.28 0.46 
uC Interact more with me around course content 15 0.27 0.46 
ait Interact more with other students around course content 25 0.24 0.44 
_8 Other (please explain) 3 1.00 0.00 
45 “How much do you agree or disagree with the following statements regarding 
{Product}? Using {Product} in this course helped me...” (1=Strongly Agree or Agree) 
| Present content more effectively 26 0.31 0.47 
2 Personalize learning in response to different students’ needs 25 0.52 0.51 
8 Reduce time spent assessing student learning 26 Om2 0.33 
_4 Reduce the time spent on mechanical teaching tasks such as recording grades 26 0.08 Or 
mS) Reduce time spent in class 26 0.04 0.20 
m6 Better monitor individual student progress 26 0.50 OS 
ats Keep better track of whole-class progress 26 0.31 0.47 
8} Increase student engagement with the content of my course 26 0.73 0.45 
ue) Use class time for more interactive activities 26 ORS: 0.37 
10) Reduce time spent preparing the course at the start of the term 15 0.13 0.35 
Ni Support my students in deeper learning 26 0.58 0.50 
2 Other 2 1.00 0.00 
46 “Did you experience any usability and technical problems when using {Product} in 26 0.69 0.47 
your course this term (spring 2017)?” (1=Yes) 
47 “Which of these usability and technical problems did you experience when using 
{Product} in your course this term?” (1=Yes) 
ll Problems with logging or signing in (either you or your students) 18 0.67 0.49 
2 Incompatibilities with existing computer hardware or software (e.g., browser 18 Omit 0.38 
incompatibilities) 
_8 Incompatibilities with our campus systems, such as the Learning Management or 18 Oma ONs2 
Student Information System 
4 Incompatibilities with other online resources 18 0.06 0.24 
mS) Lack of technical or help desk support needed from my campus 18 Onli 0.38 
6 Lack of technical or help desk support needed from {Grantee} 18 0.22 0.43 
it Confusing design of dashboard or system interface 18 0.67 0.49 
8 Other 18 0.39 0.50 
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Table B-3. Cerego, Product (n=26, 78% response rate), (Cont'd.) 


Q# Question Text n Mean SD 
48 “Which, if any, of these problems did you experience when using {Product} in your 
course this term (spring 2017)?” (1=Yes) 
a Lack of alignment with course goals 23 0.39 0.50 
ee Time required to learn new online system 23 0.48 0.51 
_3 Student resistance to using system 23 0.48 OFSal 
_4 Students did not use {Product} the way it was intended (i.e., attempting to 23 0.26 0.45 
succeed by exploiting properties of the system rather than learning the material) 
& Did not work well within course schedule or allotted class time 28} Ons 0.34 
= Other, please explain Ve Onl 0.33 
cs None of these nla 0.00 0.00 
49 “Overall, how satisfied have you been when using {Product} in your course this term 26 0.69 0.47 
(spring 2017)?” (1=Highly or Moderately Satisfied) 
5il “Do you plan to use {Product} in your course again?” 
Yes, in much the same way | used it this term 7 0.27 0.45 
Yes, but in a more limited way 6) 0.19 0.40 
Yes, and to a greater extent than | used it this term 3 On? 0.33 
No 8 0.31 0.47 
Not sure 3 Onl 0.33 
oy “How likely are you to recommend {Product} to a friend or colleague teaching the 25 5.60 PINS) 
same course you do?” (0 to 10) 
Table B-4. CogBooks, Product (n=20, 80% response rate) 
Q# Question Text n Mean SD 
1 Are you currently using PRODUCT in any of your courses? 8 1.00 0.00 
3a “About how many students are enrolled in this section of {Course}?” 
35 or fewer 10 0.67 0.49 
36-99 3 0.2 0.41 
100-249 1 0.07 0.26 
250-499 1 0.07 0.26 
500 or more 0 0.00 0.00 
3b “About how many students are enrolled in this section of {Course}?” 
35 or fewer 2 0.67 0.58 
36-99 1 0.33 0.58 
100-249 0 0.00 0.00 
250-499 0 0.00 0.00 
500 or more 0) 0.00 0.00 
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Table B-4. CogBooks, Product (n=20, 80% response rate), (Cont'd.) 


Q# Question Text 
3c “About how many students are enrolled in this section of {Course}?” 
35 or fewer 
36-99 
100-249 
250-499 


500 or more 


SRI Education 


(<< es) B) 


S|) oO) & | &| © 


3d “About how many students are enrolled in this section of {Course}?” 
35 or fewer 
36-99 
100-249 
250-499 


500 or more 


Qo eo) eo) oe | © 


3e “About how many students are enrolled in this section of {Course}?” 
35 or fewer 
36-99 
100-249 
250-499 
500 or more 


oe) oo) oe) © 


Sf “About how many students are enrolled in this section of {Course}?” 
35 or fewer 
36-99 
100-249 
250-499 
500 or more 


Qa) |e | © 


3g “About how many students are enrolled in this section of {Course}?” 
35 or fewer 
36-99 
100-249 
250-499 
500 or more 


oS) oS | oe |) © 


Sh “About how many students are enrolled in this section of {Course}?” 
35 or fewer 
36-99 
100-249 
250-499 


500 or more 


So oe |o&| © 
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Table B-4. CogBooks, Product (n=20, 80% response rate), (Cont'd.) 


SRI Education 


Q# Question Text Mean SD 
3i “About how many students are enrolled in this section of {Course}?” 
35 or fewer 0 
36-99 0 
100-249 0 
250-499 0 
500 or more ) 
3j “About how many students are enrolled in this section of {Course}?” 
35 or fewer ) 
36-99 0 
100-249 0 
250-499 0 
500 or more 0 
4 Is this your first time using PRODUCT in COURSE? 
Is this your first time using PRODUCT in COURSE? 13 0.62 0.51 
Is this your first time using PRODUCT in COURSE? 0 
Is this your first time using PRODUCT in COURSE? 0 
Is this your first time using PRODUCT in COURSE? 0 
Is this your first time using PRODUCT in COURSE? 0 
Is this your first time using PRODUCT in COURSE? 0 
6 Have you taught this course before without using {Product}? (1=Yes) 14 1.00 0.00 
5a Did you use another courseware product previously? (1=Yes) 0 0.00 0.00 
Did you teach this course with PRODUCT in a previous term? 8 0.50 0.53 
For how many terms have you used PRODUCT in the past? 
1 term 2 0.50 0.58 
2 terms 1 0.25 0.50 
3 terms 1 225 0.50 
4 terms 0 0.00 0.00 
5+ terms 0) 0.00 0.00 
10 Have you ever taught courses with online or blended learning components before 13 0.62 0.51 
using PRODUCT? 
an “Last Fall, were you teaching...” (1=Yes, Full time) slat eo 0.40 
12 “Last Fall, was your position...” 
ml Tenure track faculty 4 0.36 0.50 
cs Lecturer or Instructor 6 0.55 0.52 
= Adjunct professor, lecturer, or instructor 1 0.09 0.30 
Ie “Did you have a choice to use {Product} for your course this term (spring 2017)?” 8 0.50 ORDS 
(1=Yes) 
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Table B-4. CogBooks, Product (n=20, 80% response rate), (Cont'd.) 


Q# Question Text n Mean SD 
14 “What motivated you to choose {Product}?” (1=Yes) 
i Interest in exploring online teaching and learning 4 ORS 0.50 
2 Thought {Product} would help with course management or organization 4 0.75 0.50 
8 Thought teaching with {Product} would take less of my time 4 0.50 0.58 
A Desire for a more flexible teaching schedule 4 0.25 0.50 
mS Opportunity for professional development 4 0.50 0.58 
6 Received an attractive incentive 4 0.00 0.00 
if This was the only online learning tool available to me for this course 4 0.00 0.00 
ES To be consistent with other courses/sections 4 0.25 0.50 
9 Desire for a more convenient way to communicate with students 4 0.25 0.50 
10 Expectation that my students would be more engaged 4 1.00 0.00 
ll Expectation that my students would receive more individualized learning 4 1.00 0.00 
12 Encouragement from upper levels of your institution (for example, your 4 0.00 0.00 
department or college) to use {Product} 
18 Wanted to same my students the cost of a textbook 2, 0.50 0.71 
14 Other (please explain) 4 0.00 
1 “To what extent did using {Product} change the way you...” (1=Some Change or 
Major Change) 
= Planned this course 16 0.81 0.40 
2 Presented the material for this course 16 0.94 0.25 
_3 Interacted with students in this course 16 ORK) 0.45 
_4 Evaluated students in this course 16 0.81 0.40 
16 “Did you receive training or professional development this term (spring 2017) on 12 0.67 0.49 
how to use or integrate {product} into your course to improve student learning?” 
(1=Yes) 
ilk “How many hours of training did you receive this term?” 10 4.20 3.91 
19 “How satisfied were you with the training you received this term?” (1=Moderately or 8 1.00 0.00 
Highly Satisfied) 
20 “Was training for {Product} offered this term (spring 2017)?” (1=Yes) 4 0.00 0.00 
2 “Briefly indicate your reasons for not participating in the training for {Product} this 
year.” (1=Yes) 
i Training did not align with my schedule @) 
2 Training offered in inconvenient location or format O 
8) Training took too much time O 
_4 Training occurred before | was hired 0) 
6) | took the training previously O 
6 Did not feel the need for training 5) 0.00 
aif Other, please describe. 0 
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Table B-4. CogBooks, Product (n=20, 80% response rate), (Cont'd.) 


Q# Question Text n Mean SD 


22 “The last time you taught your course without {Product}, did you receive any 7 0.57 OSS 
training or professional development on how to design and implement instruction to 
improve student learning?” (1=Yes) 


23 “How many hours of training did you receive for your course on how to design and 4 4.50 athe 
implement instruction and to improve student learning when not using {Product}?” 


24 “When preparing for your course this spring, how much time did you spend 9 8.22 O52 
(in hours) learning how to use {Product}, in both formal training and individual 
preparation? Do not include any time spent learning the product before fall 2016. 


Zo “When preparing for your course this spring, how much time did you spend (in 8 23.88 18.33 
hours) integrating {Product} into your course (e. g., adapting assignments, adapting 
format of content or lesson delivery, adapting grading, etc.)? Do not include any 
time spent learning the product before fall 2016.” 


26 “How prepared did you feel to use {Product} this term (spring 2017)?” (1=Very or 20 ONZ5 0.44 
Adequately Prepared) 


2, “Have you received support for using {Product} outside of formal training this term 16 0.88 0.34 
(spring 2017?” (1=Yes) 
28 “From which of these sources have you received support for using {Product} this term? (1=Yes) 
mal {Product} support staff/technical assistance 14 0.93 0.27 
2 Next Generation Courseware Challenge project leader on my campus 14 Ofs7/ OFS 
_é Peers/colleagues also using {Product} 14 0.64 0.50 
_4 Technical (IT) support staff on my campus 14 0.36 0.50 
6 Instructional design staff on my campus 14 0.50 0.52 
6 Someone else from my department 14 0.21 0.43 
mc Other (please specify) 14 0.07 0.27 
29 “How helpful was this support in enabling you to use {Product} in your course this term (spring 2017)?” 
(1=Very Helpful or Helpful) 
= {Product} support staff/technical assistance iS 0.92 0.28 
2 Next Generation Courseware Challenge project leader on my campus 8 1.00 0.00 
_3 Peers/colleagues also using {Product} 9 1.00 0.00 
_4 Technical (IT) support staff on my campus 4 1.00 0.00 
mo) Instructional design staff on my campus i 1.00 0.00 
3 Someone else from my department 8 1.00 0.00 
if Other (please specify) 1 1.00 0.00 
30 “Which of the following best describes how {Product} is/was used in your course this term (spring 2017)?” 
Substitute for printed text as core course component 5 0.25 0.44 
Substitute for some lecture/class meeting time O 0.00 0.00 
Supplemental source of course content 3 ORTS OF 
Skills practice environment | 0.05 O22 
Formative assessment system 0 0.00 0.00 
Flipping the classroom (i.e., using the product to cover content in advance so 8 0.40 0.50 
that class time could be used for more active activities) 
Fully online delivery of course 3 Om 0.37 
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Table B-4. CogBooks, Product (n=20, 80% response rate), (Cont'd.) 


Q# Question Text n Mean SD 
31 “What proportion of the content in your course is being delivered using {Product}?” 
0% to 24% 0 0.00 0.00 
25% to 49% 3 0.20 0.41 
50% to 79% 6 0.40 0.51 
80% to 100% 6 0.40 0.51 
82 “Before the start of your course this term (spring 2017), how many hours would you 4 25.00 12.91 
estimate you spent planning your instruction and student assignments?” 
38a “Think about a typical week teaching your course this term (spring 2017). How 
many hours did you spend on each of the following course-related activities during 
that week?” 
Presenting course content and leading course instruction hours 4 4.25 1.26 
Interacting with students (including office hours, email correspondence, etc.) 4 4.50 0.58 
Evaluating or providing written feedback on student work 4 3.75 0.50 
Course administration (e.g. entering grades, technology assistance, etc.) 4 6925 4.03 
Revising your plan for instruction and student assignments 4 5.00 1.41 
34 “How many hours did you spend planning instruction and student assignments 
before the course started for {Course} with and without using {Product}?” 
14 With {Product} 10 SOOM a2 62 
2a Without using {Product} 10 20.10 15.84 
35 “During a typical week, how many hours did/do you spend performing the following 
tasks for {Course} with and without using {Product}?” 
ia With using {Product}: Presenting course content and leading course instruction slat 5.18 4.42 
hours 
12 With using {Product}: Interacting with students (including office hours, email slat 4.73 2.61 
correspondence, etc.) 
18) With using {Product}: Evaluating or providing written feedback on student work 14 5.09 ZS 
1_4 With using {Product}: Course administration (e.g. entering grades, technology lit 3.36 1.36 
assistance, etc.) 
1 With using {Product}: Revising your plan for instruction and student assignments ileal 3.64 PAK) 
2 || Without using {Product}: Presenting course content and leading course slat 6.18 4.35 
instruction hours 
22 Without using {Product}: Interacting with students (including office hours, email slat B20 5.00 
correspondence, etc.) 
28} Without using {Product}: Evaluating or providing written feedback on student 11 6.45 6.90 
work 
2.4 Without using {Product}: Course administration (e.g. entering grades, technology 14 4.91 4.89 
assistance, etc.) 
2a5 Without using {Product}: Revising your plan for instruction and student 14 5.64 4.01 
assignments 
36 “Overall, how would you describe the impact of using {Product} on the time you 15 0.47 O12 
spend on your course?” (1= Major or Minor Time Saver) 
SIG How important is the ability to print content to use PRODUCT? 8 0.63 O2 
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Table B-4. CogBooks, Product (n=20, 80% response rate), (Cont'd.) 


Q# Question Text n Mean SD 
38 Who pays for printing? 
College pays for printing 1 0.20 0.45 
Students pays for printing 4 0.80 0.45 
Other pays for printing 0 0.00 0.00 
39 Approximately how many pages per student are printed over the entire course? 
0-50 pages 2 0.67 0.58 
51-100 pages O 0.00 0.00 
101-150 pages 1 0.33 0.58 
151-200 pages O 0.00 0.00 
200+ pages O 0.00 0.00 
40 “When your course is taught at your institution without {Product}, are textbooks or 1S 1.00 0.00 


other materials generally required or recommended for students?” (1=Yes) 


41 “What is the approximate cost of text and other materials for your course when 
teaching it without {Product}?” (1=Less than $25, 7=More than $275) 


Less than $25 0 0.00 0.00 
$26-$75 2 Oncmmn Orcs 
$76-$125 6 0.40 0.54 
$126-$175 3 0.20 0.44 
$176-$225 3 0.20 0.41 
$226-$275 0 0.00 0.00 
More than $275 1 0.07 0.26 
Don’t know 0 0.00 0.00 
42 “Is your course staffed differently when using {Product}?” 
1 Yes, more staff 7 0.47 0.52 
Yes, less staff 1 0.07 0.26 
fe: No, same staff it 0.47 0.52 
a Don’t Know O 0.00 0.00 
43 “How frequently do you use each of the following instructor-oriented features of 
{Product}?” (1=Used All or Some of the Time) 
1 Used online gradebook to keep track of individual students’ performance 18 0.67 0.49 
Ei Used dashboards to track progress of the class as a whole 20 OMS 0.44 
i Used dashboard to check individual students’ progress in understanding 20 0.70 0.47 
concepts and content 
ee Used dashboard to check when individual students last logged in to {Product} 20 0.50 0.51 
mS) Used {Product} student performance data to refer individual students to specific 19 ORS) FSi 
content or problem activities 
6 Adjusted the structure or sequencing of course content in {Product} for some students 19 0.42 FSi 
lt Used {Product} to communicate with students 20 0.35 0.49 
8 Used {Product} to make class materials available to students sl 0.65 0.49 
9 Identifying common misconceptions so that | could address them during class time 20 0.80 0.41 
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Table B-4. CogBooks, Product (n=20, 80% response rate), (Cont'd.) 


Q# Question Text n Mean SD 
44 “How much do you agree or disagree with the following statements? Using 
{Product} helps students...” (1=Strongly Agree or Agree) 
1 Better understand and remember course content 20 ORAS 0.44 
2 Better analyze and apply course content 20 0.75 0.44 
6 Complete more complex assignments that demonstrate their understanding of 18 0.56 @rSil 
course content 
_4 Stay more engaged in the course 20 0.85 0.37 
5) Reduce costs associated with the course (e.g., purchasing a textbook) 18 0.67 0.49 
6 Interact more with me around course content 1s 0.93 0.26 
Ly Interact more with other students around course content 20 0.60 0.50 
_8 Other (please explain) 2 0.50 Oval 
45 “How much do you agree or disagree with the following statements regarding 
{Product}? Using {Product} in this course helped me...” (1=Strongly Agree or Agree) 
| Present content more effectively 20 0.75 0.44 
2 Personalize learning in response to different students’ needs 19 0.84 0.37 
3 Reduce time spent assessing student learning 20 Oro OFDal 
4 Reduce the time spent on mechanical teaching tasks such as recording grades 19 Om 0.42 
6) Reduce time spent in class 19 ORSZ, 0.48 
6 Better monitor individual student progress 20 0.65 0.49 
wif Keep better track of whole-class progress 20 0.80 0.41 
8 Increase student engagement with the content of my course 20 0.80 0.41 
9) Use class time for more interactive activities 20 0.60 0.50 
10 Reduce time spent preparing the course at the start of the term iS) 0.47 O%S2 
I Support my students in deeper learning 20 0.60 0.50 
12 Other 1 1.00 0.00 
46 “Did you experience any usability and technical problems when using {Product} in 16 0.81 0.40 
your course this term (spring 2017)?” (1=Yes) 
47 “Which of these usability and technical problems did you experience when using 
{Product} in your course this term?” (1=Yes) 
| Problems with logging or signing in (either you or your students) 13 ORat 0.44 
2 Incompatibilities with existing computer hardware or software (e.g., browser 13 Ont 0.44 
incompatibilities) 
_8 Incompatibilities with our campus systems, such as the Learning Management or ls) 0.08 0.28 
Student Information System 
4 Incompatibilities with other online resources 18 ORS 0.38 
© Lack of technical or help desk support needed from my campus ills) 0.00 0.00 
6 Lack of technical or help desk support needed from {Grantee} ils) 0.23 0.44 
LE Confusing design of dashboard or system interface 13 0.38 0.51 
8 Other 12 0.08 0.29 
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Table B-4. CogBooks, Product (n=20, 80% response rate), (Cont'd.) 


Q# Question Text n Mean SD 
48 “Which, if any, of these problems did you experience when using {Product} in your 
course this term (spring 2017)?” (1=Yes) 
a Lack of alignment with course goals 20 0.25 0.44 
_2 Time required to learn new online system 20 0.15 0.37 
_3 Student resistance to using system 20 ORES 0.37 
_4 Students did not use {Product} the way it was intended (i.e., attempting to 20 0.40 0.50 
succeed by exploiting properties of the system rather than learning the material) 
ad Did not work well within course schedule or allotted class time 20 O10) Orsi 
= Other, please explain 14 0.00 0.00 
* None of these 14 0.21 0.48 
49 “Overall, how satisfied have you been when using {Product} in your course this term 20 Tells 0.93 
(spring 2017)?” (1=Highly or Moderately Satisfied) 
5il “Do you plan to use {Product} in your course again?” 
Yes, in much the same way | used it this term 11 0.55 OFS 
Yes, but in a more limited way 8 O25 0.44 
Yes, and to a greater extent than | used it this term 4 0.20 0.41 
No 0) 
Not sure 0 
52 “How likely are you to recommend {Product} to a friend or colleague teaching the 20 7.65 ZE2S 
same course you do?” (0 to 10) 
Table B-5. Lumen, Product (n=108, 70% response rate) 
Q# Question Text n NeXT a) B) 
1 Are you currently using PRODUCT in any of your courses? 48 1.00 0.00 
8a “About how many students are enrolled in this section of {Course}?” 
35 or fewer 54 0.92 0.28 
36-99 5) OOS MI Or23 
100-249 0 
250-499 @) 
500 or more 0 
3b “About how many students are enrolled in this section of {Course}?” 
35 or fewer 14 0.88 0.34 
36-99 2 0.13 0.34 
100-249 @) 
250-499 0 
500 or more 0 
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Table B-5. Lumen, Product (n=108, 70% response rate) (Cont'd.) 


Q# Question Text 


86 “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 
500 or more 


SRI Education 


0.67 0.52 
0.33 0.52 


ey te) | ©) | | 5s 


3d “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 
500 or more 


0.75 0.50 
0.25 0.50 


eS |e >) & 


8e “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 


500 or more 


0.75 0.50 
0.25 0.50 


ey) ee) | = | i) 


Sf “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 


500 or more 


0.50 0.71 
0.50 0.71 


oO oOo oO + + 


3g “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 


500 or more 


eS) ©) @) oe |& 


Sh “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 
500 or more 


eye | ee | & 
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Table B-5. Lumen, Product (n=108, 70% response rate) (Cont'd.) 


SRI Education 


Q# Question Text Mean SD 
3i “About how many students are enrolled in this section of {Course}?” 
35 or fewer 0 
36-99 0 
100-249 0 
250-499 0 
500 or more 0 
3} “About how many students are enrolled in this section of {Course}?” 
35 or fewer ) 
36-99 0 
100-249 0 
250-499 0 
500 or more 0 
4 Is this your first time using PRODUCT in COURSE? 
Is this your first time using PRODUCT in COURSE? 97 0.48 0.50 
Is this your first time using PRODUCT in COURSE? 18 0.60 0.51 
Is this your first time using PRODUCT in COURSE? 0.67 0.58 
Is this your first time using PRODUCT in COURSE? 1 1.00 0.00 
Is this your first time using PRODUCT in COURSE? 
Is this your first time using PRODUCT in COURSE? 
5 Have you taught this course before without using {Product}? (1=Yes) 15 1.00 0.00 
5a Did you use another courseware product previously? (1=Yes) 0 
8 Did you teach this course with PRODUCT in a previous term? 48 ORAS 0.45 
9 For how many terms have you used PRODUCT in the past? 
1 term 6 0.17 0.38 
2 terms 14 0.31 0.47 
3 terms 10 0.29 0.46 
4 terms 4 0.11 0.32 
5+ terms 4 0.11 0.32 
10 Have you ever taught courses with online or blended learning components before 97 0.74 0.44 
using PRODUCT? 
Tl “Last Fall, were you teaching...” (1=Yes, Full time) 49 49 OFSi 
12 “Last Fall, was your position...” 
=| Tenure track faculty 2X6) 0.47 0.50 
2 Lecturer or Instructor 3 0.06 0.24 
8 Adjunct professor, lecturer, or instructor 23 0.47 0.50 
13 “Did you have a choice to use {Product} for your course this term (spring 2017)?” 85 0.36 0.48 
(1=Yes) 
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Table B-5. Lumen, Product (n=108, 70% response rate) (Cont'd.) 


Q# Question Text n Mean SD 
14 “What motivated you to choose {Product}?” (1=Yes) 
| Interest in exploring online teaching and learning 31 Onan 0.43 
2 Thought {Product} would help with course management or organization 31 0.81 0.40 
me) Thought teaching with {Product} would take less of my time 31 0.35 0.49 
_4 Desire for a more flexible teaching schedule 31 0.26 0.44 
= Opportunity for professional development 31 OL00 0.43 
=6 Received an attractive incentive 31 0.23 0.43 
i This was the only online learning tool available to me for this course 31 0.06 0:25 
_8 To be consistent with other courses/sections 31 0.26 0.44 
9 Desire for a more convenient way to communicate with students 61 OHSS 0.51 
m0 Expectation that my students would be more engaged 31 0.84 0.37 
alli Expectation that my students would receive more individualized learning 31 0.90 0.30 
2 Encouragement from upper levels of your institution (for example, your 31 0.39 0.50 
department or college) to use {Product} 
Hil Wanted to same my students the cost of a textbook 1 1.00 0.00 
_14 Other (please explain) 31 0.23 0.43 
1) “To what extent did using {Product} change the way you...” (1=Some Change or 
Major Change) 
al Planned this course 98 0.59 0.49 
2 Presented the material for this course 98 0.74 0.44 
_3 Interacted with students in this course 98 0.76 0.43 
_4 Evaluated students in this course 98 0.82 0.39 
16 “Did you receive training or professional development this term (spring 2017) on 95 0.71 0.46 
how to use or integrate {product} into your course to improve student learning?” 
(1=Yes) 
allie “How many hours of training did you receive this term?” 13 6.69 7.49 
19 “How satisfied were you with the training you received this term?” (1=Moderately or 67 0.84 0.37 
Highly Satisfied) 
20 “Was training for {Product} offered this term (spring 2017)?” (1=Yes) aif 0.52 OFSal 
21 “Briefly indicate your reasons for not participating in the training for {Product} this 
year.” (1=Yes) 
1 Training did not align with my schedule 13 0.38 0.51 
2 Training offered in inconvenient location or format 18) Ons 0.38 
_3 Training took too much time 13 ORS 0.38 
_4 Training occurred before | was hired 13 0.15 0.38 
5) | took the training previously ake) 0.67 0.49 
=6 Did not feel the need for training 62 0.13 0.34 
OG Other, please describe. 13 0.08 0.28 
22 “The last time you taught your course without {Product}, did you receive any 4 0.00 0.00 


training or professional development on how to design and implement instruction to 
improve student learning?” (1=Yes) 
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Table B-5. Lumen, Product (n=108, 70% response rate) (Cont'd.) 


Q# Question Text n Mean SD 
23 “How many hours of training did you receive for your course on how to design and O 
implement instruction and to improve student learning when not using {Product}?” 
24 “When preparing for your course this spring, how much time did you spend 93 11.70 =10.26 
(in hours) learning how to use {Product}, in both formal training and individual 
preparation? Do not include any time spent learning the product before fall 2016. 
25) “When preparing for your course this spring, how much time did you spend (in 90 13.99 13.25 
hours) integrating {Product} into your course (e. g., adapting assignments, adapting 
format of content or lesson delivery, adapting grading, etc.)? Do not include any 
time spent learning the product before fall 2016.” 
26 “How prepared did you feel to use {Product} this term (spring 2017)?” (1=Very or 108 0.78 0.42 
Adequately Prepared) 
if “Have you received support for using {Product} outside of formal training this term 97 0.62 0.49 
(spring 2017?” (1=Yes) 
28 “From which of these sources have you received support for using {Product} this term? (1=Yes) 
1 {Product} support staff/technical assistance 60 0.83 0.38 
2 Next Generation Courseware Challenge project leader on my campus 60 0.27 0.45 
_3 Peers/colleagues also using {Product} 60 0.47 0.50 
4 Technical (IT) support staff on my campus 60 OF 0.49 
=o) Instructional design staff on my campus 60 Oss 0.48 
_6 Someone else from my department 60 0.28 0.45 
_7 Other (please specify) 60 Ont 0.32 
29 “How helpful was this support in enabling you to use {Product} in your course this term (spring 2017)”?” 
(1=Very Helpful or Helpful) 
ml {Product} support staff/technical assistance 50 0.98 0.14 
2 Next Generation Courseware Challenge project leader on my campus 16 0.94 O25 
_3 Peers/colleagues also using {Product} 28 0.89 0.31 
4 Technical (IT) support staff on my campus 22. 0.82 0.39 
5) Instructional design staff on my campus 20 1.00 0.00 
6 Someone else from my department 16 0.94 0.25 
if Other (please specify) 2 1.00 0.00 
30 “Which of the following best describes how {Product} is/was used in your course this term (spring 2017)?” 
Substitute for printed text as core course component 107 0.35 0.48 
Substitute for some lecture/class meeting time 107 0.02 0.14 
Supplemental source of course content OW OO OrZS 
Skills practice environment 107 OLOS MORI, 
Formative assessment system 107 0.06 0.23 
Flipping the classroom (i.e., using the product to cover content in advance so 107 0.01 0.10 
that class time could be used for more active activities) 
Fully online delivery of course Ome O24 8 O250) 
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Table B-5. Lumen, Product (n=108, 70% response rate) (Cont'd.) 


SRI Education 


Q# Question Text Mean SD 
Sill “What proportion of the content in your course is being delivered using {Product}?” 
0% to 24% 5 0.08 0.28 
25% to 49% 0.02 0.13 
50% to 79% WV 0.29 0.46 
80% to 100% 36 0.61 0.49 
32 “Before the start of your course this term (spring 2017), how many hours would you 10 9.60 7.78 
estimate you spent planning your instruction and student assignments?” 
33a “Think about a typical week teaching your course this term (spring 2017). How 
many hours did you spend on each of the following course-related activities during 
that week?” 
Presenting course content and leading course instruction hours 10 6.30 4.99 
Interacting with students (including office hours, email correspondence, etc.) 10 7.60 6.70 
Evaluating or providing written feedback on student work 10 4.30 1.83 
Course administration (e.g. entering grades, technology assistance, etc.) 10 4.10 2.42 
Revising your plan for instruction and student assignments 10 5.00 1.41 
34 “How many hours did you spend planning instruction and student assignments 
before the course started for {Course} with and without using {Product}?” 
1 With {Product} 15 10.60 6.85 
Pal Without using {Product} 18 15.00 11.88 
35 “During a typical week, how many hours did/do you spend performing the following 
tasks for {Course} with and without using {Product}?” 
ileal With using {Product}: Presenting course content and leading course instruction 15 5.00 2.98 
hours 
2 With using {Product}: Interacting with students (including office hours, email 15 4.73 2.09 
correspondence, etc.) 
lame With using {Product}: Evaluating or providing written feedback on student work 15 5.00 3.51 
ila With using {Product}: Course administration (e.g. entering grades, technology 1S 8.07 2.34 
assistance, etc.) 
ile) With using {Product}: Revising your plan for instruction and student assignments iS Chie) e116) 
| Without using {Product}: Presenting course content and leading course VS SrOlG 2.94 
instruction hours 
2a. Without using {Product}: Interacting with students (including office hours, email 15 4.87 2.29 
correspondence, etc.) 
Za8 Without using {Product}: Evaluating or providing written feedback on student 1s SEAS B28) 
work 
2.4 Without using {Product}: Course administration (e.g. entering grades, technology US 3.13 2.26 
assistance, etc.) 
2S Without using {Product}: Revising your plan for instruction and student 15 4.27 5.02 
assignments 
36 “Overall, how would you describe the impact of using {Product} on the time you 24 0.58 0.50 
spend on your course?” (1= Major or Minor Time Saver) 
87 How important is the ability to print content to use PRODUCT? 13 0.69 0.48 
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Table B-5. Lumen, Product (n=108, 70% response rate) (Cont'd.) 


Q# Question Text n Mean SD 
38 Who pays for printing? 


College pays for printing 3 Os 0.50 
Students pays for printing 5 0.56 0.53 
Other pays for printing 1 Oba 0.33 


39 Approximately how many pages per student are printed over the entire course? 


0-50 pages 3 0.75 0.50 
51-100 pages 0 
101-150 pages 0) 
151-200 pages 0 
200+ pages 1 0.25 0.50 
40 “When your course is taught at your institution without {Product}, are textbooks or 24 0.83 0.38 
other materials generally required or recommended for students?” (1=Yes) 
41 “What is the approximate cost of text and other materials for your course when 
teaching it without {Product}?” (1=Less than $25, 7=More than $275) 
Less than $25 4 0.20 30.41 
$26-$75 2 OnOMOLsit 
$76-$125 6 0.30 0.47 
$126-$175 2 O10 0.c1 
$176-$225 5 0.25 0.44 
$226-$275 1 0.05 0.22 
More than $275 O 
Don’t know 0) 
42 ‘Is your course staffed differently when using {Product}?” 
1 Yes, more staff 3 0.13 0.34 
2 Yes, less staff 1 0.04 0.20 
_3 No, same staff 19 0.79 0.41 
4 Don’t Know 1 0.04 0.20 
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Table B-5. Lumen, Product (n=108, 70% response rate) (Cont'd.) 


Q# Question Text n Mean SD 
43 “How frequently do you use each of the following instructor-oriented features of 
{Product}?” (1=Used All or Some of the Time) 
zal Used online gradebook to keep track of individual students’ performance 103 0.85 OFS5 
2 Used dashboards to track progress of the class as a whole 99 0.47 0.50 
8 Used dashboard to check individual students’ progress in understanding concepts 99 0.64 0.48 
and content 
_4 Used dashboard to check when individual students last logged in to {Product} 99 0.45 0.50 
mS Used {Product} student performance data to refer individual students to specific 100 0.61 0.49 
content or problem activities 
6 Adjusted the structure or sequencing of course content in {Product} for some 95 0.24 0.43 
students 
a Used {Product} to communicate with students 102 0.70 0.46 
= Used {Product} to make class materials available to students 102 0.79 0.41 
= Identifying common misconceptions so that | could address them during class 98 0.34 0.48 
time 
44 “How much do you agree or disagree with the following statements? Using {Product} 
helps students...” (1=Strongly Agree or Agree) 
1 Better understand and remember course content 107 0.54 0.50 
2 Better analyze and apply course content 107 0.54 0.50 
_& Complete more complex assignments that demonstrate their understanding of 106 0.43 0.50 
course content 
_4 Stay more engaged in the course 107 ORS2 0.50 
ES Reduce costs associated with the course (e.g., purchasing a textbook) 107 0.49 0.50 
=o Interact more with me around course content 58 0.86 0.35 
fl Interact more with other students around course content 107 0.32 0.47 
8 Other (please explain) 10 0.50 OF5s 
45 “How much do you agree or disagree with the following statements regarding 
{Product}? Using {Product} in this course helped me...” (1=Strongly Agree or Agree) 
ail Present content more effectively 107 0.61 0.49 
ee Personalize learning in response to different students’ needs 106 0.58 0.50 
8} Reduce time spent assessing student learning 107 0.64 0.48 
_4 Reduce the time spent on mechanical teaching tasks such as recording grades 107 0.63 0.49 
_& Reduce time spent in class 106 0.32 0.47 
_6 Better monitor individual student progress 107 0.69 0.46 
mG Keep better track of whole-class progress 107 0.57 0.50 
8 Increase student engagement with the content of my course 107 0.46 0.50 
8) Use class time for more interactive activities 106 0.36 0.48 
_10 Reduce time spent preparing the course at the start of the term 58 0.43 0.50 
1 Support my students in deeper learning 107 0.54 0.50 
2 Other 20 0.70 0.47 
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Table B-5. Lumen, Product (n=108, 70% response rate) (Cont'd.) 


Q# Question Text n Mean SD 
46 “Did you experience any usability and technical problems when using {Product} in 97 0.46 0.50 
your course this term (spring 2017)?” (1=Yes) 
47 “Which of these usability and technical problems did you experience when using 
{Product} in your course this term?” (1=Yes) 
= Problems with logging or signing in (either you or your students) 45 0.27 0.45 
2 Incompatibilities with existing computer hardware or software (e.g., browser 45 0.42 0.50 
incompatibilities) 
_§ Incompatibilities with our campus systems, such as the Learning Management or 45 0.31 0.47 
Student Information System 
4 Incompatibilities with other online resources 45 0.22 0.42 
us Lack of technical or help desk support needed from my campus 45 0.20 0.40 
_& Lack of technical or help desk support needed from {Grantee} 45 0.24 0.43 
at Confusing design of dashboard or system interface 45 0.42 0.50 
_8 Other 44 0.45 0.50 
48 “Which, if any, of these problems did you experience when using {Product} in your 
course this term (spring 2017)?” (1=Yes) 
el Lack of alignment with course goals 86 0.10 0.31 
2 Time required to learn new online system 86 0.30 0.46 
nS Student resistance to using system 86 0.27 0.45 
_4 Students did not use {Product} the way it was intended (i.e., attempting to 86 0.35 0.48 
succeed by exploiting properties of the system rather than learning the material) 
ES Did not work well within course schedule or allotted class time 86 0.07 0.26 
a Other, please explain 61 0.08 0.28 
None of these Sy 0.11 0.31 
49 “Overall, how satisfied have you been when using {Product} in your course this term 107 0.88 0.54 
(spring 2017)?” (1=Highly or Moderately Satisfied) 
oil “Do you plan to use {Product} in your course again?” 
Yes, in much the same way | used it this term 41 0.39 0.49 
Yes, but in a more limited way 32 0.30 0.46 
Yes, and to a greater extent than | used it this term 14 Ons 0.34 
No 10 0.10 0.29 
Not sure 8 0.08 0.27 
52 “How likely are you to recommend {Product} to a friend or colleague teaching the 106 205 8.05 


same course you do?” (0 to 10) 
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Table B-6. Rice University, Product (n=164, 80% response rate) 


Q# Question Text n Mean SD 

1 Are you currently using PRODUCT in any of your courses? 146 1.00 0.00 

3a “About how many students are enrolled in this section of {Course}?” 
35 or fewer 158 0.96 ORS 
36-99 3 0.02 0.13 
100-249 2 0.01 0.11 
250-499 1 0.01 0.08 
500 or more 0 

3b “About how many students are enrolled in this section of {Course}?” 
35 or fewer 29 0.97 0.18 
36-99 0 
100-249 1 OL0/ORIS 
250-499 0 
500 or more ) 

3c “About how many students are enrolled in this section of {Course}?” 
35 or fewer 10 1.00 0.00 
36-99 0 
100-249 0 
250-499 0 
500 or more 0 

3d “About how many students are enrolled in this section of {Course}?” 
35 or fewer 3 1.00 0.00 
36-99 0 
100-249 0 
250-499 0 
500 or more 0 

3e “About how many students are enrolled in this section of {Course}?” 
35 or fewer 2 1.00 0.00 
36-99 0 
100-249 0 
250-499 0 
500 or more 0 
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Table B-6. Rice University, Product (n=164, 80% response rate) (Cont'd.) 


Q# Question Text n i (¥2 s) B) 


St “About how many students are enrolled in this section of {Course}?” 
35 or fewer 1.00 0.00 

36-99 

100-249 


250-499 


ey) @) ee) &) th 


500 or more 


3g “About how many students are enrolled in this section of {Course}?” 


ef 
an 
ro) 
ro) 


35 or fewer 0.00 
36-99 
100-249 


250-499 


gy GJ o&| © 


500 or more 


3h “About how many students are enrolled in this section of {Course}?” 


=e 
= 
fo) 
S 


35 or fewer 0.00 
36-99 
100-249 


250-499 


S| |) of) © 


500 or more 


3i “About how many students are enrolled in this section of {Course}?” 


at 
x 
ro) 
S 


35 or fewer 0.00 
36-99 
100-249 


250-499 


oro) 2] © 


500 or more 


3j “About how many students are enrolled in this section of {Course}?” 


at 
ae 
ro) 
iS 


35 or fewer 0.00 
36-99 
100-249 


250-499 


e| | & | © 


500 or more 
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Table B-6. Rice University, Product (n=164, 80% response rate) (Cont'd.) 


Q# Question Text n Mean SD 

4 Is this your first time using PRODUCT in COURSE? 
Is this your first time using PRODUCT in COURSE? 145 0.58 0.50 
Is this your first time using PRODUCT in COURSE? 16 0.50 0.52 
Is this your first time using PRODUCT in COURSE? 5) 0.60 OFS 
Is this your first time using PRODUCT in COURSE? 1 1.00 0.00 
Is this your first time using PRODUCT in COURSE? 0 
Is this your first time using PRODUCT in COURSE? 

5 Have you taught this course before without using {Product}? (1=Yes) 18 1.00 0.00 

5a Did you use another courseware product previously? (1=Yes) 8 0.25 0.46 
Did you teach this course with PRODUCT in a previous term? 68 0.47 0.50 


For how many terms have you used PRODUCT in the past? 


1 term 35 0.51 0.50 
2 terms 20 0:29 0.46 
3 terms 9 0.13 0.34 
4 terms 2 0.03 Onlin 
5+ terms 2 0.03 Onlin 
10 Have you ever taught courses with online or blended learning components before 145 0.67 0.47 
using PRODUCT? 
14 “Last Fall, were you teaching...” (1=Yes, Full time) 150 .82 0.39 
12 “Last Fall, was your position...” 
I Tenure track faculty 100 0.64 0.48 
= Lecturer or Instructor 29 0.18 0.39 
zm Adjunct professor, lecturer, or instructor 28 0.18 0.38 
13 ae have a choice to use {Product} for your course this term (spring 2017)?” Si 0.96 0.19 
=Yes 
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Table B-6. Rice University, Product (n=164, 80% response rate) (Cont'd.) 


SRI Education 


Q# Question Text n Mean SD 
14 “What motivated you to choose {Product}?” (1=Yes) 
i Interest in exploring online teaching and learning lou 0.70 0.46 
= Thought {Product} would help with course management or organization iow 0.44 OFSO) 
_3 Thought teaching with {Product} would take less of my time IS On25 0.43 
_4 Desire for a more flexible teaching schedule 151 0.07 0.26 
mS Opportunity for professional development 151 0.23 0.42 
6 Received an attractive incentive 151 0.05 0.21 
if This was the only online learning tool available to me for this course ilou Oo OF32 
as} To be consistent with other courses/sections 151 0.09 0.29 
9 Desire for a more convenient way to communicate with students 151 0.25 0.44 
10 Expectation that my students would be more engaged 151 0.86 Ors5 
14 Expectation that my students would receive more individualized learning 151 0.83 0.38 
IZ Encouragement from upper levels of your institution (for example, your 151 0.09 0.28 
department or college) to use {Product} 
13 Wanted to same my students the cost of a textbook 10 0.80 0.42 
14 Other (please explain) ito 0.34 0.47 
1S “To what extent did using {Product} change the way you...” (1=Some Change or 
Major Change) 
ll Planned this course 157 0.24 0.43 
_2 Presented the material for this course 157 0.20 0.40 
_8 Interacted with students in this course SH 0.19 0.39 
_4 Evaluated students in this course 157 0.38 0.49 
16 “Did you receive training or professional development this term (spring 2017) on how 164 0.21 0.41 
to use or integrate {product} into your course to improve student learning?” (1=Yes) 
alla “How many hours of training did you receive this term?” 6 1.83 ORAS 
19 “How satisfied were you with the training you received this term?” (1=Moderately or 34 0.91 0.29 
Highly Satisfied) 
20 “Was training for {Product} offered this term (spring 2017)?” (1=Yes) 130 0.30 0.46 
21 “Briefly indicate your reasons for not participating in the training for {Product} this 
year.” (1=Yes) 
il Training did not align with my schedule 39 0.26 0.44 
_2 Training offered in inconvenient location or format 389 0.08 0.16 
_3 Training took too much time 38 0.18 0.39 
_4 Training occurred before | was hired 39 0.00 
_& | took the training previously 38 0.29 0.46 
6 Did not feel the need for training 39 0.67 0.48 
ff Other, please describe. 38 0.08 0.27 
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Table B-6. Rice University, Product (n=164, 80% response rate) (Cont'd.) 


Q# Question Text n Mean SD 


22 “The last time you taught your course without {Product}, did you receive any training 11 0.18 0.40 
or professional development on how to design and implement instruction to improve 
student learning?” (1=Yes) 


23 “How many hours of training did you receive for your course on how to design and 2 7.00 4.24 
implement instruction and to improve student learning when not using {Product}?” 


24 “When preparing for your course this spring, how much time did you spend (in hours) 146 2.45 2.56 
earning how to use {Product}, in both formal training and individual preparation? Do 
not include any time spent learning the product before fall 2016. 


25 “When preparing for your course this spring, how much time did you spend (in hours) 146 Teer ee) 
integrating {Product} into your course (e. g., adapting assignments, adapting format 

of content or lesson delivery, adapting grading, etc.)? Do not include any time spent 

earning the product before fall 2016.” 


26 “How prepared did you feel to use {Product} this term (spring 2017)?” (1=Very or 164 0.66 0.48 
Adequately Prepared) 


27 “Have you received support for using {Product} outside of formal training this term sfey4 OFSil 0.46 
(spring 2017?” (1=Yes) 
28 “From which of these sources have you received support for using {Product} this term? (1=Yes) 
1 {Product} support staff/technical assistance 49 0.92 0.28 
Le Next Generation Courseware Challenge project leader on my campus 49 0.10 0.31 
ms) Peers/colleagues also using {Product} 49 0.16 0.37 
_4 Technical (IT) support staff on my campus 49 0.02 0.14 
=5 Instructional design staff on my campus 49 0.06 0.24 
_6 Someone else from my department 49 0.04 0.20 
_7 Other (please specify) 49 0.06 0.24 
29 “How helpful was this support in enabling you to use {Product} in your course this term (spring 2017)”?” (1=Very 
Helpful or Helpful) 
_ {Product} support staff/technical assistance 45 0.91 0.29 
B2. Next Generation Courseware Challenge project leader on my campus 5 0.60 OFS5 
a8 Peers/colleagues also using {Product} 8 0.88 0.35 
4 Technical (IT) support staff on my campus 1 1.00 0.00 
eS) Instructional design staff on my campus 3 0.67 0.58 
_6 Someone else from my department 1 0.00 
Ba? Other (please specify) @ 1.00 0.00 
30 “Which of the following best describes how {Product} is/was used in your course this term (spring 2017)?” 
Substitute for printed text as core course component 44 0.25 0.43 
Substitute for some lecture/class meeting time 1 0.01 0.08 
Supplemental source of course content 34 0.21 0.41 
Skills practice environment 58 0.35 0.48 
Formative assessment system ils) 0.08 0.27 
Flipping the classroom (i.e., using the product to cover content in advance so that 14 0.09 0.28 
class time could be used for more active activities) 
Fully online delivery of course 3 0.02 0.13 
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Table B-6. Rice University, Product (n=164, 80% response rate) (Cont'd.) 


Q# Question Text n Mean SD 
Sil “What proportion of the content in your course is being delivered using {Product}?” 
0% to 24% 100 0.61 0.49 
25% to 49% Dt 0.16 0.37 
50% to 79% 13 0.08 Oi 
80% to 100% 24 OR CHOFSS 
32 “Before the start of your course this term (spring 2017), how many hours would you 0 


estimate you spent planning your instruction and student assignments?” 


88a = “Think about a typical week teaching your course this term (spring 2017). How many 
hours did you spend on each of the following course-related activities during that 


week?” 
Presenting course content and leading course instruction hours O 
Interacting with students (including office hours, email correspondence, etc.) O 
Evaluating or providing written feedback on student work 0 
Course administration (e.g. entering grades, technology assistance, etc.) 0 
Revising your plan for instruction and student assignments O 
34 “How many hours did you spend planning instruction and student assignments 
before the course started for {Course} with and without using {Product}?” 
1 With {Product} 18 Wierasy hehe} 
2 || Without using {Product} 18 11.28 8.64 
385 “During a typical week, how many hours did/do you spend performing the following 
tasks for {Course} with and without using {Product}?” 
Jel With using {Product}: Presenting course content and leading course instruction 18 6.83 8.51 
hours 
1-2 With using {Product}: Interacting with students (including office hours, email 18 5 6il 3.38 
correspondence, etc.) 
1_3 With using {Product}: Evaluating or providing written feedback on student work 18 4.06 3.24 
1_4 With using {Product}: Course administration (e.g. entering grades, technology 18 4.33 3.79 
assistance, etc.) 
1 With using {Product}: Revising your plan for instruction and student assignments 18 Ona 2.42 
2 Without using {Product}: Presenting course content and leading course instruction 18 6.83 8.51 
hours 
De Without using {Product}: Interacting with students (including office hours, email 18 5.33 Oro 
correspondence, etc.) 
2_3 Without using {Product}: Evaluating or providing written feedback on student work 18 4.11 3.25 
2.4 Without using {Product}: Course administration (e.g. entering grades, technology 18 4.06 3.35 
assistance, etc.) 
mS) Without using {Product}: Revising your plan for instruction and student 18 2.67 aera 
assignments 
36 “Overall, how would you describe the impact of using {Product} on the time you 18 Ol 0.32 
spend on your course?” (1= Major or Minor Time Saver) 
37 How important is the ability to print content to use PRODUCT? O 
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Table B-6. Rice University, Product (n=164, 80% response rate) (Cont'd.) 


Q# Question Text n Mean SD 
38 Who pays for printing? 


College pays for printing O 
Students pays for printing 0 
Other pays for printing 0 


39 Approximately how many pages per student are printed over the entire course? 


0-50 pages O 
51-100 pages 0 
101-150 pages 0) 
151-200 pages 0 
200+ pages 0 
40 “When your course is taught at your institution without {Product}, are textbooks or 18 .89 FO? 
other materials generally required or recommended for students?” (1=Yes) 
41 “What is the approximate cost of text and other materials for your course when 
teaching it without {Product}?” (1=Less than $25, 7=More than $275) 
Less than $25 i 0.44 0.51 
$26-$75 1 0.06 0.25 
$76-$125 4 O25 ee 46 
$126-$175 0 0.00 0.00 
$176-$225 2 O13 ors 
$226-$275 0 0.00 0.00 
More than $275 2 0.13 0.34 
Don’t know 0 0.00 0.00 
42 “Is your course staffed differently when using {Product}?” 
= Yes, more staff 2 0.11 0.32 
a Yes, less staff ) 0.00 0.00 
4 No, same staff 15 0.83 0.38 
4 Don’t Know 1 0.06 0.24 
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Table B-6. Rice University, Product (n=164, 80% response rate) (Cont'd.) 


Q# Question Text n Mean SD 
43 “How frequently do you use each of the following instructor-oriented features of 
{Product}?” (1=Used All or Some of the Time) 
3 Used online gradebook to keep track of individual students’ performance lS 0.59 0.49 
es Used dashboards to track progress of the class as a whole 161 0.45 0.50 


Used dashboard to check individual students’ progress in understanding concepts 162 0.49 0.50 
and content 


2 Used dashboard to check when individual students last logged in to {Product} 161 0.24 0.43 

e Used {Product} student performance data to refer individual students to specific 160 0.22 0.41 
content or problem activities 

_6 Adjusted the structure or sequencing of course content in {Product} for some 156 0.09 0.29 
students 

we Used {Product} to communicate with students 150 0.04 0.20 

a Used {Product} to make class materials available to students 155 0.39 0.49 


Identifying common misconceptions so that | could address them during class time 162 0.30 0.46 


44 “How much do you agree or disagree with the following statements? Using {Product} 

helps students...” (1=Strongly Agree or Agree) 
= Better understand and remember course content 164 0.72 0.45 
= Better analyze and apply course content 164 0.59 0.49 
a Complete more complex assignments that demonstrate their understanding of 164 0.40 0.49 

course content 
4 Stay more engaged in the course 164 0.73 0.45 
eS Reduce costs associated with the course (e.g., purchasing a textbook) 164 0.35 0.48 
a6 Interact more with me around course content 164 0.82 0.39 
aff Interact more with other students around course content 164 0.26 0.44 
8) Other (please explain) oil 0.32 0.48 
45 “How much do you agree or disagree with the following statements regarding 

{Product}? Using {Product} in this course helped me...” (1=Strongly Agree or Agree) 
=| Present content more effectively 163 0.38 0.49 
_2 Personalize learning in response to different students’ needs 164 0.40 0.49 
_3 Reduce time spent assessing student learning 164 07 0.45 
4 Reduce the time spent on mechanical teaching tasks such as recording grades 164 Of22 0.42 
5 Reduce time spent in class 164 0.10 0.31 
6 Better monitor individual student progress 164 0.59 0.49 
eft Keep better track of whole-class progress 164 OR55 0.50 
ms) Increase student engagement with the content of my course 163 0.67 0.47 
-@ Use class time for more interactive activities 164 0.42 0.50 
1 Reduce time spent preparing the course at the start of the term 163 0.14 ORS 
mali Support my students in deeper learning 164 0.65 0.48 
12 Other 38 0.47 0.51 
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Table B-6. Rice University, Product (n=164, 80% response rate) (Cont'd.) 


Q# Question Text n Mean SD 
46 “Did you experience any usability and technical problems when using {Product} in 156 0.45 0.50 
your course this term (spring 2017)?” (1=Yes) 
47 “Which of these usability and technical problems did you experience when using 
{Product} in your course this term?” (1=Yes) 
=| Problems with logging or signing in (either you or your students) 71 0.70 0.46 
2 Incompatibilities with existing computer hardware or software (e.g., browser 71 0.25 0.44 
incompatibilities) 
_8 Incompatibilities with our campus systems, such as the Learning Management or 71 0.23 0.42 
Student Information System 
_4 Incompatibilities with other online resources al 0.06 OF23 
_& Lack of technical or help desk support needed from my campus aa 0.21 0.41 
GC Lack of technical or help desk support needed from {Grantee} 71 OMS 0.34 
_tf Confusing design of dashboard or system interface ia 0.35 0.48 
fe Other faa 0.49 0.50 
48 “Which, if any, of these problems did you experience when using {Product} in your 
course this term (spring 2017)?” (1=Yes) 
al Lack of alignment with course goals 163 0.16 0.37 
2 Time required to learn new online system 163 0.19 0.39 
8) Student resistance to using system 163 0.43 0.50 
ft Students did not use {Product} the way it was intended (i.e., attempting to succeed 163 0.44 0.50 


by exploiting properties of the system rather than learning the material) 


Z Did not work well within course schedule or allotted class time 163 0.05 0.22 
m= Other, please explain 133 0.20 0.40 
= None of these 126 0.05 0.21 
49 “Overall, how satisfied have you been when using {Product} in your course this term 163 0.85 0.36 
(spring 2017)?” (1=Highly or Moderately Satisfied) 
‘OM “Do you plan to use {Product} in your course again?” 
Yes, in much the same way | used it this term 41 O25 0.44 
Yes, but in a more limited way 84 0.52 OFSO 
Yes, and to a greater extent than | used it this term 14 0.09 0.28 
No 17 OOO: 
Not sure 7 0.04 0.20 
62 “How likely are you to recommend {Product} to a friend or colleague teaching the 163 6.26 2.40 


same course you do?” (0 to 10) 
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Table B-7. Smart Sparrow, Product (n=100, 80% response rate) 


Q# Question Text n Mean i) 
1 Are you currently using PRODUCT in any of your courses? 56 1.00 0.00 


8a “About how many students are enrolled in this section of {Course}?” 
35 or fewer 74 0.94 0.25 

36-99 5 0.06 0.25 

100-249 0 
250-499 0 
0 


500 or more 


3b “About how many students are enrolled in this section of {Course}?” 


or) 
= 
ro) 
iS) 


35 or fewer 1 0.00 
36-99 
100-249 


250-499 


je)| xe) | er) ©@ 


500 or more 


Se “About how many students are enrolled in this section of {Course}?” 
35 or fewer 1.00 0.00 

36-99 

100-249 


250-499 


ea!) @}] © fi 


500 or more 


3d “About how many students are enrolled in this section of {Course}?” 
35 or fewer 1.00 0.00 

36-99 

100-249 


250-499 


oe ©) ©) ©) © 


500 or more 


8e “About how many students are enrolled in this section of {Course}?” 


= 
ro) 
ro) 


35 or fewer 0.00 
36-99 
100-249 


250-499 


je] @ | Ce) eo 


500 or more 
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Table B-7. Smart Sparrow, Product (n=100, 80% response rate), (Cont'd.) 


Q# Question Text 


Sf “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 


500 or more 


SRI Education 


NY (=¥1 9 es) B) 


Qe) ea) & ©) © 


3g “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 


500 or more 


eo oe oe) eS © 


3h “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 


500 or more 


er) ey) eo) wo] XS 


3i “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 


500 or more 


Se @& 2 & 


3j “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 


500 or more 


OIe@) FO) eso 
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Table B-7. Smart Sparrow, Product (n=100, 80% response rate), (Cont'd.) 


SRI Education 


Q# Question Text Mean SD 
4 Is this your first time using PRODUCT in COURSE? 
Is this your first time using PRODUCT in COURSE? Ut 0.74 0.44 
Is this your first time using PRODUCT in COURSE? 5) 1.00 0.00 
Is this your first time using PRODUCT in COURSE? 1 1.00 0.00 
Is this your first time using PRODUCT in COURSE? ) 
Is this your first time using PRODUCT in COURSE? 0 
Is this your first time using PRODUCT in COURSE? 0 
5) Have you taught this course before without using {Product}? (1=Yes) 83 0.91 0.29 
5a Did you use another courseware product previously? (1=Yes) 0 
Did you teach this course with PRODUCT in a previous term? 156 0.23 0.43 
For how many terms have you used PRODUCT in the past? 
1 term 9 0.69 0.48 
2 terms 2 OAS: 0.38 
3 terms 1 0.08 0.28 
4 terms 0 
5+ terms 1 0.08 0.28 
10 Have you ever taught courses with online or blended learning components before aa 0.58 0.50 
using PRODUCT? 
rltak “Last Fall, were you teaching...” (1=Yes, Full time) 67 .B4 O87 
12 “Last Fall, was your position...” 
1 Tenure track faculty 31 0.46 0.50 
x Lecturer or Instructor 25) 0.37 0.49 
a Adjunct professor, lecturer, or instructor alist 0.16 0:37 
is) “Did you have a choice to use {Product} for your course this term (spring 2017)?” Gs 0.79 0.41 


(1=Yes) 


Next Generation Courseware Challenge Evaluation 


133 


SRI Education 


Table B-7. Smart Sparrow, Product (n=100, 80% response rate), (Cont'd.) 


Q# Question Text n Mean SD 
14 “What motivated you to choose {Product}?” (1=Yes) 
1 Interest in exploring online teaching and learning 58 0.79 0.41 
2 Thought {Product} would help with course management or organization 58 0.43 0.50 
8 Thought teaching with {Product} would take less of my time 58 0.12 0.33 
_4 Desire for a more flexible teaching schedule 58 OR OFSS 
§ Opportunity for professional develooment 58 0.59 0.50 
6 Received an attractive incentive 58 On 0.33 
_if This was the only online learning tool available to me for this course 58 0.07 0.26 
=S To be consistent with other courses/sections 58 0.10 0.31 
9 Desire for a more convenient way to communicate with students 58 0.19 0.40 
_10 Expectation that my students would be more engaged 58 0.95 0.22 
17 Expectation that my students would receive more individualized learning 58 0.84 0.37 
12 Encouragement from upper levels of your institution (for example, your 58 0.36 0.48 
department or college) to use {Product} 
Bali Wanted to same my students the cost of a textbook 5 0.80 0.45 
mail Other (please explain) 58 0.24 0.43 
18 “To what extent did using {Product} change the way you...” (1=Some Change or 
Major Change) 
ml Planned this course 89 0.61 0.49 
_2 Presented the material for this course 88 0.58 0.50 
_é Interacted with students in this course 89 ORSS 0.50 
_4 Evaluated students in this course 89 0.67 0.47 
16 “Did you receive training or professional development this term (spring 2017) on 84 0.60 0.49 
how to use or integrate {product} into your course to improve student learning?” 
(1=Yes) 
lv “How many hours of training did you receive this term?” 23 5.39 8.07 
19 “How satisfied were you with the training you received this term?” (1=Moderately or 50 0.92 O27, 
Highly Satisfied) 
20 “Was training for {Product} offered this term (spring 2017)?” (1=Yes) 383 0.48 OrSil 
21 “Briefly indicate your reasons for not participating in the training for {Product} this 
year.” (1=Yes) 
mz Training did not align with my schedule 16 0.63 0.50 
2 Training offered in inconvenient location or format 16 0.19 0.40 
8 Training took too much time 16 0.31 0.48 
_4 Training occurred before | was hired 16 0.06 0.25 
zo) | took the training previously 13 0.15 0.38 
6 Did not feel the need for training 34 0.18 0.39 
if Other, please describe. Us 0.27 0.46 
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Table B-7. Smart Sparrow, Product (n=100, 80% response rate), (Cont'd.) 


Q# Question Text n Mean SD 
22 “The last time you taught your course without {Product}, did you receive any 20 0.50 FSi 
training or professional development on how to design and implement instruction to 
improve student learning?” (1=Yes) 
23 “How many hours of training did you receive for your course on how to design and 9 11 12.30 
implement instruction and to improve student learning when not using {Product}?” 
24 “When preparing for your course this spring, how much time did you spend 70 10.70 10.31 
(in hours) learning how to use {Product}, in both formal training and individual 
preparation? Do not include any time spent learning the product before fall 2016. 
25 “When preparing for your course this spring, how much time did you spend (in 70 17.36 14.47 
hours) integrating {Product} into your course (e. g., adapting assignments, adapting 
format of content or lesson delivery, adapting grading, etc.)? Do not include any 
time spent learning the product before fall 2016.” 
26 “How prepared did you feel to use {Product} this term (spring 2017)?” (1=Very or 100 0.53 0.50 
Adequately Prepared) 
Oi “Have you received support for using {Product} outside of formal training this term 86 0.81 0.39 
(spring 2017?” (1=Yes) 
28 “From which of these sources have you received support for using {Product} this term? (1=Yes) 
al {Product} support staff/technical assistance 70 0.97 Onl 
2 Next Generation Courseware Challenge project leader on my campus 70 Om 0.38 
rs Peers/colleagues also using {Product} 70 Orb 0.50 
4 Technical (IT) support staff on my campus 70 0.19 0.39 
§ Instructional design staff on my campus 70 Onl 0.38 
8 Someone else from my department 70 0.21 0.41 
ff Other (please specify) 70 0.07 0.26 
29 “How helpful was this support in enabling you to use {Product} in your course this term (spring 201 7)”?” 
(1=Very Helpful or Helpful) 
aa {Product} support staff/technical assistance 68 0.85 0.36 
2 Next Generation Courseware Challenge project leader on my campus 12 0.92 0.29 
rs Peers/colleagues also using {Product} 40 0.85 0.36 
_4 Technical (IT) support staff on my campus ake) 0.77 0.44 
& Instructional design staff on my campus 12 1.00 0.00 
6 Someone else from my department 15 0.67 0.49 
mig Other (please specify) 3 0.33 0.58 
30 “Which of the following best describes how {Product} is/was used in your course this term (spring 2017)?” 
Substitute for printed text as core course component 20 0.20 0.40 
Substitute for some lecture/class meeting time iG 0.07 0.26 
Supplemental source of course content 42 0.42 0.50 
Skills practice environment ite 0.07 0.26 
Formative assessment system 4 0.04 0.20 
Flipping the classroom (i.e., using the product to cover content in advance so 12 Ome 0.33 
that class time could be used for more active activities) 
Fully online delivery of course 8 0.08 0.27 
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Table B-7. Smart Sparrow, Product (n=100, 80% response rate), (Cont'd.) 


SRI Education 


Q# Question Text Mean SD 
Sil “What proportion of the content in your course is being delivered using {Product}?” 
0% to 24% 29 0.37 0.49 
25% to 49% 30 0.38 0.49 
50% to 79% ali 0.14 OSS 
80% to 100% 0.11 0.32 
82 “Before the start of your course this term (spring 2017), how many hours would you 9 225 Om GrO2 
estimate you spent planning your instruction and student assignments?” 
33a “Think about a typical week teaching your course this term (spring 2017). How 
many hours did you spend on each of the following course-related activities during 
that week?” 
Presenting course content and leading course instruction hours 9 S1a/ 2.96 
Interacting with students (including office hours, email correspondence, etc.) 9 4.22 SalS) 
Evaluating or providing written feedback on student work 9 4.44 2.01 
Course administration (e.g. entering grades, technology assistance, etc.) 9 4.56 le Sil 
Revising your plan for instruction and student assignments 9 4.33 2.06 
34 “How many hours did you spend planning instruction and student assignments 
before the course started for {Course} with and without using {Product}?” 
1 With {Product} 30 16.97 12.82 
2 Without using {Product} 29 17.66 14.59 
35 “During a typical week, how many hours did/do you spend performing the following 
tasks for {Course} with and without using {Product}?” 
ileal With using {Product}: Presenting course content and leading course instruction 29 4.34 3.63 
hours 
12 With using {Product}: Interacting with students (including office hours, email 29 7.07 9.04 
correspondence, etc.) 
cle With using {Product}: Evaluating or providing written feedback on student work 30 4.00 Salli 
alee With using {Product}: Course administration (e.g. entering grades, technology 29 3.86 Sail if 
assistance, etc.) 
139 With using {Product}: Revising your plan for instruction and student assignments 28 3.50 3.14 
2) Without using {Product}: Presenting course content and leading course 29 S10 4.55 
instruction hours 
2a? Without using {Product}: Interacting with students (including office hours, email 29 6.34 6.65 
correspondence, etc.) 
2.3 Without using {Product}: Evaluating or providing written feedback on student 29 4.83 4.97 
work 
2.4 Without using {Product}: Course administration (e.g. entering grades, technology 29 3.66 3.30 
assistance, etc.) 
mS Without using {Product}: Revising your plan for instruction and student 28 8.25 2.61 
assignments 
36 “Overall, how would you describe the impact of using {Product} on the time you 38 ORG 0.49 
spend on your course?” (1= Major or Minor Time Saver) 
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Table B-7. Smart Sparrow, Product (n=100, 80% response rate), (Cont'd.) 


Q# Question Text n Mean SD 
Si How important is the ability to print content to use PRODUCT? 16 0.38 0.50 


38 Who pays for printing? 


College pays for printing 2 0.33 0.52 
Students pays for printing 2 0.33 Ola2 
Other pays for printing 2 0.33 OFZ 


39 Approximately how many pages per student are printed over the entire course? 


0-50 pages 3 ORS 0.50 
51-100 pages 0 
101-150 pages 1 0.25 0.50 
151-200 pages 0 
200+ pages 0 
40 “When your course is taught at your institution without {Product}, are textbooks or 38 1.00 0.00 
other materials generally required or recommended for students?” (1=Yes) 
41 “What is the approximate cost of text and other materials for your course when 
teaching it without {Product}?” (1=Less than $25, 7=More than $275) 
Less than $25 0 
$26-$75 5 OLE hes 
$76-$125 10 OTS 
$126-$175 7 0.19 0.40 
$176-$225 10 Oe yaks 
$226-$275 4 Ol - hel 
More than $275 0 
Don’t know 1 0.03 0.16 
42 ‘Is your course staffed differently when using {Product}?” 
1 Yes, more staff 3 0.08 0.28 
2 Yes, less staff 8 0.22 0.28 
_3 No, same staff 23 0.64 0.49 
4 Don’t Know 2 0.06 0.23 
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Table B-7. Smart Sparrow, Product (n=100, 80% response rate), (Cont'd.) 


Q# Question Text n Mean SD 
43 “How frequently do you use each of the following instructor-oriented features of 
{Product}?” (1=Used All or Some of the Time) 
1 Used online gradebook to keep track of individual students’ performance 97 0.62 0.49 
= Used dashboards to track progress of the class as a whole 98 0.47 0.50 
Pr Used dashboard to check individual students’ progress in understanding 96 0.40 0.49 
concepts and content 
= Used dashboard to check when individual students last logged in to {Product} 96 0.28 0.45 
ze Used {Product} student performance data to refer individual students to specific 98 0.19 0.40 
content or problem activities 
6 Adjusted the structure or sequencing of course content in {Product} for some 98 0.21 0.41 
students 
7 Used {Product} to communicate with students 94 0.07 0.26 
| Used {Product} to make class materials available to students 92 0.34 0.48 
ee Identifying common misconceptions so that | could address them during class 96 0.30 0.46 
time 
44 “How much do you agree or disagree with the following statements? Using 
{Product} helps students...” (1=Strongly Agree or Agree) 
il Better understand and remember course content 100 0.59 0.49 
= Better analyze and apply course content 100 0.65 0.48 
a Complete more complex assignments that demonstrate their understanding of 100 0.66 0.48 
course content 
_4 Stay more engaged in the course 100 0.59 0.49 
mS) Reduce costs associated with the course (e.g., purchasing a textbook) 99 0.38 0.49 
6 Interact more with me around course content 78 0.64 0.48 
a Interact more with other students around course content 99 0.40 0.49 
8 Other (please explain) 9 0.56 0.53 
45 “How much do you agree or disagree with the following statements regarding 
{Product}? Using {Product} in this course helped me...” (1=Strongly Agree or Agree) 
1 Present content more effectively 99 OFS 0.50 
2 Personalize learning in response to different students’ needs 99 0.46 0.50 
_3 Reduce time spent assessing student learning 99 0.28 0.45 
_4 Reduce the time spent on mechanical teaching tasks such as recording grades 99 0.24 0.43 
5) Reduce time spent in class 98 0.26 0.44 
6 Better monitor individual student progress 100 0.40 0.49 
_if Keep better track of whole-class progress 99 0.39 0.49 
ts Increase student engagement with the content of my course 100 0.68 0.47 
_9 Use class time for more interactive activities 97 0.47 0.50 
_10 Reduce time spent preparing the course at the start of the term 78 Ona 0.38 
ai Support my students in deeper learning 99 0.68 0.47 
12 Other 18 0.56 0.51 
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Table B-7. Smart Sparrow, Product (n=100, 80% response rate), (Cont'd.) 


Q# Question Text n Mean SD 
46 “Did you experience any usability and technical problems when using {Product} in 89 0.85 0.36 
your course this term (spring 2017)?” (1=Yes) 
47 “Which of these usability and technical problems did you experience when using 
{Product} in your course this term?” (1=Yes) 
al Problems with logging or signing in (either you or your students) 76 0.63 0.49 
2 Incompatibilities with existing computer hardware or software (e.g., browser 76 0.72 0.45 
incompatibilities) 
3 Incompatibilities with our campus systems, such as the Learning Management or 76 0.36 0.48 
Student Information System 
4 Incompatibilities with other online resources 76 0.12 0.33 
mS Lack of technical or help desk support needed from my campus 76 0.22 0.42 
6 Lack of technical or help desk support needed from {Grantee} ve 0.25 0.44 
au Confusing design of dashboard or system interface 76 0.54 0.50 
_8 Other 76 0.42 0.50 
48 “Which, if any, of these problems did you experience when using {Product} in your 
course this term (spring 2017)?” (1=Yes) 
_1 Lack of alignment with course goals 97 0.23 0.42 
2 Time required to learn new online system 97 0.42 0.50 
3 Student resistance to using system 97 0.60 0.49 
4 Students did not use {Product} the way it was intended (i.e., attempting to 97 0.40 0.49 
succeed by exploiting properties of the system rather than learning the material) 
a Did not work well within course schedule or allotted class time 97 0.16 (Si 
_ Other, please explain 86 0.22 0.42 
= None of these 86 0.01 0.11 
49 “Overall, how satisfied have you been when using {Product} in your course this term 100 0.92 0.82 
(spring 2017)?” (1=Highly or Moderately Satisfied) 
51 “Do you plan to use {Product} in your course again?” 
Yes, in much the same way | used it this term 28 0.28 0.45 
Yes, but in a more limited way 34 0.34 0.48 
Yes, and to a greater extent than | used it this term 12 0.12 0.33 
No alt 0.11 0.31 
Not sure 15 ORS 0.36 
52 “How likely are you to recommend {Product} to a friend or colleague teaching the 100 6.21 2.90 


same course you do?” (0 to 10) 
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Table B-8. Stanford, Product (n=100, 79% response rate) 


Q# Question Text n Mean SD 
1 Are you currently using PRODUCT in any of your courses? 75 1.00 0.00 


8a “About how many students are enrolled in this section of {Course}?” 
35 or fewer ia) 1.00 0.00 
36-99 
100-249 
250-499 


500 or more 


3b “About how many students are enrolled in this section of {Course}?” 
85 or fewer 1.00 0.00 

36-99 

100-249 


250-499 


| Te) | eo | oS) ©) 


500 or more 


SG “About how many students are enrolled in this section of {Course}?” 
35 or fewer 
36-99 
100-249 
250-499 


OO) 1) @) @ 


500 or more 


3d “About how many students are enrolled in this section of {Course}?” 
35 or fewer 
36-99 
100-249 
250-499 


©) ©) ©) es oS 


500 or more 


3e “About how many students are enrolled in this section of {Course}?” 
35 or fewer 
36-99 
100-249 
250-499 


ey) ey | ey eo) 2 


500 or more 
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Table B-8. Stanford, Product (n=100, 79% response rate), (Cont'd.) 


Q# Question Text n Mean SD 


of “About how many students are enrolled in this section of {Course}?” 
35 or fewer 
36-99 
100-249 
250-499 


eS) 2) oS 


500 or more 


3g “About how many students are enrolled in this section of {Course}?” 


35 or fewer 
36-99 
100-249 
250-499 


ey eee 2) 


500 or more 


3h “About how many students are enrolled in this section of {Course}?” 
35 or fewer 
36-99 
100-249 
250-499 


©. © ©) © © 


500 or more 


3i “About how many students are enrolled in this section of {Course}?” 
35 or fewer 
36-99 
100-249 
250-499 


ey) ~~) 2) eo) = 


500 or more 


3} “About how many students are enrolled in this section of {Course}?” 
35 or fewer 
36-99 
100-249 
250-499 


@)| ©) ©) | eo 


500 or more 
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Table B-8. Stanford, Product (n=100, 79% response rate), (Cont'd.) 


Q# Question Text n Mean SD 
4 Is this your first time using PRODUCT in COURSE? 
Is this your first time using PRODUCT in COURSE? 100 0.39 0.49 
Is this your first time using PRODUCT in COURSE? 13 0.23 0.44 
Is this your first time using PRODUCT in COURSE? 5) 0.60 O55) 
Is this your first time using PRODUCT in COURSE? 0 
Is this your first time using PRODUCT in COURSE? 0 
Is this your first time using PRODUCT in COURSE? 0 
5 Have you taught this course before without using {Product}? (1=Yes) 7 0.43 OR5S 
5a Did you use another courseware product previously? (1=Yes) O 0.00 0.00 
8 Did you teach this course with PRODUCT in a previous term? 1) Oa 0.50 
9 For how many terms have you used PRODUCT in the past? 
1 term 6 0.14 0.35 
2 terms G 0.16 0.37 
3 terms 2 0.05 0.21 
4 terms 4 0.09 0.29 
5+ terms 24 0.56 0.50 
10 Have you ever taught courses with online or blended learning components before 100 0.64 0.48 
using PRODUCT? 
11 “Last Fall, were you teaching...” (1=Yes, Full time) 74 zi 0.49 
12 “Last Fall, was your position...” 
x Tenure track faculty 39 OFS 0.50 
S Lecturer or Instructor 8 Oral Orsi 
6} Adjunct professor, lecturer, or instructor a7 0.36 0.48 
1é ae have a choice to use {Product} for your course this term (spring 2017)?” Ui 0.62 0.49 
=Yes 
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Table B-8. Stanford, Product (n=100, 79% response rate), (Cont'd.) 


Q# Question Text n Mean SD 
14 “What motivated you to choose {Product}?” (1=Yes) 
| Interest in exploring online teaching and learning 48 0.69 0.47 
2 Thought {Product} would help with course management or organization 48 0.58 0.50 
6 Thought teaching with {Product} would take less of my time 48 0.19 0.39 
_4 Desire for a more flexible teaching schedule 48 0.19 0.39 
_§ Opportunity for professional development 48 0.42 0.50 
6 Received an attractive incentive 48 0.02 0.14 
_if This was the only online learning tool available to me for this course 48 0.08 0.28 
=o To be consistent with other courses/sections 48 0.40 0.49 
9 Desire for a more convenient way to communicate with students 48 0.25 0.44 
_10 Expectation that my students would be more engaged 48 Onan 0.42 
ali Expectation that my students would receive more individualized learning 48 ORRS 0.44 
all2 Encouragement from upper levels of your institution (for example, your 48 O825 0.44 
department or college) to use {Product} 
mals Wanted to same my students the cost of a textbook O 
_14 Other (please explain) 48 0.40 0.49 
15) “To what extent did using {Product} change the way you...” (1=Some Change or 
Major Change) 
-1 Planned this course 100 0.76 0.43 
2 Presented the material for this course 100 0.76 0.43 
mS Interacted with students in this course 100 0.63 0.49 
_4 Evaluated students in this course 99 0.69 0.47 
16 “Did you receive training or professional development this term (spring 2017) on ala 0.48 0.50 
how to use or integrate {product} into your course to improve student learning?” 
(1=Yes) 
Wie “How many hours of training did you receive this term?” 12 6.25 5.08 
19 “How satisfied were you with the training you received this term?” (1=Moderately Si 0.97 0.16 
or Highly Satisfied) 
20 “Was training for {Product} offered this term (spring 2017)?” (1=Yes) 38 0.03 0.16 
21 “Briefly indicate your reasons for not participating in the training for {Product} this 
year.” (1=Yes) 
il Training did not align with my schedule 1 0.00 0.00 
2 Training offered in inconvenient location or format 1 0.00 0.00 
_3 Training took too much time 1 0.00 0.00 
_4 Training occurred before | was hired 1 0.00 0.00 
mS) | took the training previously 1 1.00 0.00 
6 Did not feel the need for training 26 0.00 0.00 
mg Other, please describe. 1 0.00 0.00 
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Q# Question Text Mean SD 
22 “The last time you taught your course without {Product}, did you receive any 3 0.67 0.58 
training or professional development on how to design and implement instruction 
to improve student learning?” (1=Yes) 
23 “How many hours of training did you receive for your course on how to design and 2 4.00 1.41 
implement instruction and to improve student learning when not using {Product}?” 
24 “When preparing for your course this spring, how much time did you spend 95 11.638 11.64 
(in hours) learning how to use {Product}, in both formal training and individual 
preparation? Do not include any time spent learning the product before fall 2016. 
25 “When preparing for your course this spring, how much time did you spend 95 19:0, illseo2 
(in hours) integrating {Product} into your course (e. g., adapting assignments, 
adapting format of content or lesson delivery, adapting grading, etc.)? Do not 
include any time spent learning the product before fall 2016.” 
26 “How prepared did you feel to use {Product} this term (spring 2017)?” (1=Very or 100 0.80 0.40 
Adequately Prepared) 
27 “Have you received support for using {Product} outside of formal training this 98 0.54 0.50 
term (spring 2017?” (1=Yes) 
28 “From which of these sources have you received support for using {Product} this term? (1=Yes) 
| {Product} support staff/technical assistance 53 0.68 0.47 
_2 Next Generation Courseware Challenge project leader on my campus 53 0.11 0.32 
_3 Peers/colleagues also using {Product} 53 0.81 0.39 
4 Technical (IT) support staff on my campus 5S 0.25 0.43 
6 Instructional design staff on my campus 53 O23 0.42 
_6 Someone else from my department 53 0.38 0.49 
tf Other (please specify) 5S) 0.09 0.30 
29 “How helpful was this support in enabling you to use {Product} in your course this term (spring 2017)’?” 
(1=Very Helpful or Helpful) 
1 {Product} support staff/technical assistance 36 0.86 0.35 
2 Next Generation Courseware Challenge project leader on my campus 6 1.00 0.00 
_3 Peers/colleagues also using {Product} 43 0.86 0.35 
_4 Technical (IT) support staff on my campus 12 0.50 0.52 
mS) Instructional design staff on my campus 14 0.73 0.47 
_6 Someone else from my department 19 0.84 OMS 
ait Other (please specify) 2 1.00 0.00 
30 “Which of the following best describes how {Product} is/was used in your course this term (spring 2017)?” 
Substitute for printed text as core course component 52 Ol52 0.50 
Substitute for some lecture/class meeting time 2 0.02 0.14 
Supplemental source of course content 22 0.22 0.42 
Skills practice environment 9 0.09 0.29 
Formative assessment system 2 0.02 0.14 
Flipping the classroom (i.e., using the product to cover content in advance so 9 0.09 0.29 
that class time could be used for more active activities) 
Fully online delivery of course 4 0.04 0.20 
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Table B-8. Stanford, Product (n=100, 79% response rate), (Cont'd.) 


Q# Question Text n NV (-¥- 1 eS) B) 
Gil “What proportion of the content in your course is being delivered using 
{Product}?” 

0% to 24% 15 0.20 0.40 
25% to 49% 17 0.23 0.42 
50% to 79% 23 0.31 0.46 
80% to 100% 20 O27 0.45 

32 “Before the start of your course this term (spring 2017), how many hours would 20 {45 W207 


you estimate you spent planning your instruction and student assignments?” 


33a “Think about a typical week teaching your course this term (spring 2017). How 
many hours did you spend on each of the following course-related activities 
during that week?” 


Presenting course content and leading course instruction hours 19 6.47 2.97 
Interacting with students (including office hours, email correspondence, etc.) 19 7.74 5.69 
Evaluating or providing written feedback on student work 19 4.26 Wage 
Course administration (e.g. entering grades, technology assistance, etc.) 19 6.26 5.59 
Revising your plan for instruction and student assignments 19 6.89 8.12 
34 “How many hours did you spend planning instruction and student assignments 
before the course started for {Course} with and without using {Product}?” 
ila With {Product} 3 4.67 1583 
aml Without using {Product} 3 6.33 4.16 
85 “During a typical week, how many hours did/do you spend performing the 
following tasks for {Course} with and without using {Product}?” 
aed With using {Product}: Presenting course content and leading course instruction 3 6.67 3.06 
hours 
12 With using {Product}: Interacting with students (including office hours, email 3 3.67 2.08 
correspondence, etc.) 
18 With using {Product}: Evaluating or providing written feedback on student work 3 4.00 1.00 
1_4 With using {Product}: Course administration (e.g. entering grades, technology 3 4.33 1.53 
assistance, etc.) 
dS With using {Product}: Revising your plan for instruction and student 3 Gere) 0.58 
assignments 
al Without using {Product}: Presenting course content and leading course 3 6.67 3.06 
instruction hours 
2D Without using {Product}: Interacting with students (including office hours, email 3 4.00 1.73 
correspondence, etc.) 
Das) Without using {Product}: Evaluating or providing written feedback on student 3 3.00 1.00 
work 
2_4 Without using {Product}: Course administration (e.g. entering grades, 3 Seis 1.53 


technology assistance, etc.) 


ZS Without using {Product}: Revising your plan for instruction and student 3 3.33 0.58 
assignments 


36 “Overall, how would you describe the impact of using {Product} on the time you 23 O12 0.51 
spend on your course?” (1= Major or Minor Time Saver) 
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Table B-8. Stanford, Product (n=100, 79% response rate), (Cont'd.) 


Q# Question Text n Mean SD 
OM How important is the ability to print content to use PRODUCT? 23 0.70 0.47 


38 Who pays for printing? 


College pays for printing 9 0.56 0.51 
Students pays for printing 5 0.31 0.48 
Other pays for printing 2 Ons 0.34 
39 Approximately how many pages per student are printed over the entire course? 
0-50 pages 6 0.50 0.52 
51-100 pages 2 Onn 0.39 
101-150 pages 1 0.08 0.29 
151-200 pages 1 0.08 0.29 
200+ pages 2 0.17 0.39 
40 “When your course is taught at your institution without {Product}, are textbooks or 22 Ona 0.43 
other materials generally required or recommended for students?” (1=Yes) 
41 “What is the approximate cost of text and other materials for your course when 
teaching it without {Product}?” (1=Less than $25, 7=More than $275) 
Less than $25 4 0.29 0.47 
$26-$75 0 
$76-$125 3 0.21 0.43 
$126-$175 3 O22 eee4s 
$176-$225 4 0.29 0.47 
$226-$275 0 
More than $275 0) 
Don’t know 0 
42 “Is your course staffed differently when using {Product}?” (1=Yes, more staff) 
1 Yes, more staff 5 0.26 0.45 
2 Yes, less staff 1 505 .23 
_3 No, same staff 13 0.68 0.48 
4 Don’t Know 
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Table B-8. Stanford, Product (n=100, 79% response rate), (Cont'd.) 


Q# Question Text n Mean SD 
43 “How frequently do you use each of the following instructor-oriented features of 
{Product}?” (1=Used All or Some of the Time) 
mal Used online gradebook to keep track of individual students’ performance 96 0.84 0.36 
- Used dashboards to track progress of the class as a whole 100 0.60 0.49 
= Used dashboard to check individual students’ progress in understanding 100 OlSs 0.50 
concepts and content 
Bs Used dashboard to check when individual students last logged in to {Product} 98 0.32 0.47 
e Used {Product} student performance data to refer individual students to specific 99 0.30 0.46 
content or problem activities 
_& Adjusted the structure or sequencing of course content in {Product} for some 98 0.16 0.37 
students 
me Used {Product} to communicate with students 99 0.19 0.40 
me Used {Product} to make class materials available to students 97 0.41 0.49 
ai F Identifying common misconceptions so that | could address them during class 93 0.40 0.49 
ime 
44 “How much do you agree or disagree with the following statements? Using 
{Product} helps students...” (1=Strongly Agree or Agree) 
ll Better understand and remember course content 99 0.74 0.44 
_2 Better analyze and apply course content 99 ORAS) 0.44 
_3 Complete more complex assignments that demonstrate their understanding of 99 0.54 0.50 
course content 
_4 Stay more engaged in the course 99 0.64 0.48 
ES Reduce costs associated with the course (e.g., purchasing a textbook) 99 OFS) 0.50 
6 Interact more with me around course content Tes 0.89 0.31 
a, Interact more with other students around course content 99 0.44 0.50 
8 Other (please explain) 8 0.50 ORSe 
45 “How much do you agree or disagree with the following statements regarding 
{Product}? Using {Product} in this course helped me...” (1=Strongly Agree or 
Agree) 
= Present content more effectively 99 0.69 0.47 
2 Personalize learning in response to different students’ needs 99 0.47 0.50 
_3 Reduce time spent assessing student learning 100 OS 0.50 
_4 Reduce the time spent on mechanical teaching tasks such as recording grades 100 0.51 0.50 
_& Reduce time spent in class 97 0.24 0.43 
=6 Better monitor individual student progress 100 0.64 0.48 
_f Keep better track of whole-class progress 100 0.63 0.49 
_8 Increase student engagement with the content of my course 99 0.69 0.47 
me) Use class time for more interactive activities 938 Onan 0.43 
mal©) Reduce time spent preparing the course at the start of the term 74 0.39 0.49 
1 Support my students in deeper learning 99 0.76 0.43 
12 Other 16 0.69 0.48 
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Table B-8. Stanford, Product (n=100, 79% response rate), (Cont'd.) 


Q# Question Text n Mean SD 


46 “Did you experience any usability and technical problems when using {Product} in 100 0.46 0.50 
your course this term (spring 2017)?” (1=Yes) 


47 “Which of these usability and technical problems did you experience when using 
{Product} in your course this term?” (1=Yes) 
Bil Problems with logging or signing in (either you or your students) 46 0.41 0.50 
2 Incompatibilities with existing computer hardware or software (e.g., browser 46 Oro 0.50 
incompatibilities) 
8} Incompatibilities with our campus systems, such as the Learning Management or 46 0.26 0.44 
Student Information System 
me Incompatibilities with other online resources 46 Ons 0.36 
_& Lack of technical or help desk support needed from my campus 146 0.22 0.42 
mo Lack of technical or help desk support needed from {Grantee} 46 0.11 OFS 
mh Confusing design of dashboard or system interface 46 0.37 0.49 
ES Other 43 0.42 0.50 
48 “Which, if any, of these problems did you experience when using {Product} in your 
course this term (spring 2017)?” (1=Yes) 
al Lack of alignment with course goals 87 0.24 0.43 
m2. Time required to learn new online system 87 0.24 0.43 
8} Student resistance to using system 87 0.46 0.50 
a Students did not use {Product} the way it was intended (i.e., attempting to 87 0.38 0.49 
succeed by exploiting properties of the system rather than learning the material) 
z=) Did not work well within course schedule or allotted class time 87 0.09 0.29 
ae) Other, please explain 65 Oma Orel 
» None of these 0 0.00 0.00 
49 “Overall, how satisfied have you been when using {Product} in your course this 100 0.91 0.29 
term (spring 2017)?” (1=Highly or Moderately Satisfied) 
51 “Do you plan to use {Product} in your course again?” (1=Yes) 
Yes, in much the same way | used it this term 22 On22 0.42 
Yes, but in a more limited way 58 0.59 0.50 
Yes, and to a greater extent than | used it this term 9 0.09 0.29 
No 6 0.06 0.24 
Not sure 4 0.04 0.20 
52 “How likely are you to recommend {Product} to a friend or colleague teaching the 99 C2 Pas) 


same course you do?” (0 to 10) 
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Appendix C: Impact Estimation Methods 


The dataset screening criteria for conducting an prior achievement measure); (4) at least 30 students 
impact analysis were: (1) submission of data for in each of the treatment and comparison groups; 
both a treatment group and a comparison group; (2) and (5) baseline equivalence (not more than .25 
common end-of-course outcome measure (usually standard deviation difference between treatment and 
course grades); (3) enrollment status and common comparison students on any student characteristic or 
prior achievement or pretest measure (Student’s pretest measure). 


cumulative college GPA was an acceptable common 


Table C-1. Impact Analysis Variables of Interest and Their Coding Within the Impact Analyses 


MET are] 0) (=) Coding 


Student predictor variables 


Condition 1 = treatment condition 
Gender 1 = female? 
Race/Ethnicity 1 = underrepresented population? 
Pell Status 1 = Pell-receiving or Pell-eligible° 
Age Continuous 

First Generation College Goer 1 = Yes 

Full-time Enrollment 1 = Yes 

Cumulative GPA Continuous? 
Cumulative Credits Earned Continuous? 

Prior Attempts for This Course 1 = Yes 

Common Pretest/Prior Achievement Continuous® 


Course predictor variables 


Course Duration (in weeks) Continuous 


Outcome variables 


Course grade Continuous; 4-point grade scale 
Course completion 1 = C- or better; O = otherwise 
Credits earned in the course Continuous 
Posttest measure Continuous 


a Some institutions allowed for non-binary gender responses. These institutions and responses were rare. Such responses were coded as 0.5. 
b Under Represented Population was defined as students identifying as a race/ethnicity other than Asian or White. 

c The meaning of “Pell status” varied by institution. Some institutions reported Pell receiving, some reported Pell eligibility, and others reported both. 
d These variables only included data prior to the term used for the impact analysis. 

e Other than Cumulative GPA. 
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There were two main reasons for excluding predictor 
variables from an impact analysis. First, there was 
too much missingness in the data, either because 
the institution did not report the data (e.g., first 
generation college goer) or because large numbers 
of students lacked meaningful data (e.g., prior 
achievement measures such as ACT scores). 
Second, there was insufficient variation in the data 
to allow for effective statistical modeling (e.g., all 
courses had the same course duration). 


Impact estimates for course grades were conducted 
using ordinary least squares regression. Multilevel 
modeling would have been more appropriate 

given the study designs (either modeling students 
clustered within classrooms or students clustered 
within instructors), however the structure of the 
data prevented effective multilevel modeling. Course 
grades were converted to a 4-point grade scale 

to allow for consistent interpretation of regression 
coefficients across impact analyses and to allow 

for interpretations to be meaningful in context. 

For example, a significant Treatment regression 
coefficient estimate of +0.30 could be interpreted as 
approximately a one-step increase in course grade 
(e.g., B to a B+). 


Impact estimates for course completion were 
conducted using logistic regression. Again, multilevel 
logistic regression would have been more appropriate 
given the study designs, however the structure of the 
data prevented effective multilevel modeling. These 
predictive models were reviewed for accuracy based 
on correct classification rates (the ratio of students 
correctly predicted in either condition), sensitivity 
(true positive rate), and specificity (true negative 
rate). The desired standard for these models was 

for each rate to meet an 80+% threshold. These 
models consistently failed to meet this threshold for 
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specificity, meaning too many non-completers were 
incorrectly predicted by the model to complete the 
course. Therefore, these models are not reported in 
our impact analyses. 


Impact estimates for credits earned in courses and 
for posttest measures were not conducted due 
institutions not reporting these data or (in the case of 
credits earned) there being insufficient variation in the 
data for effective modeling. 


SRI arrived at the decision of a minimum sample 
size of 30 students in each condition for conducting 
impact analyses based on the following assumptions: 


e Assume a “large” effect size (0.35 f?) 
e Assume the standard 0.05 alpha level 


e Use a non-standard 0.95 power level to more 
reliably detect a large effect size 


e Assume three predictors (condition, enrollment 
status, prior achievement measure) for the 
minimal impact model 


The resulting student-level power analysis yielded a 
requirement of 20 students in each condition. Given 
the expectation that some courseware may not, in 
fact, have a “large” effect size and that we expected 
to in practice have a greater number of predictors, 
this number was increased to 30 students in each 
condition. 


It is important to note that a student-level power 
analysis is not in alignment with the study designs of 
the impact analyses. Conditions were determined at 
the classroom and/or teacher levels, and therefore 
an accurate power analysis would require clustering 
students within either classrooms or teachers as 
appropriate. Without accounting for teacher-level 

or classroom-level effects in the modeling (which 

as noted previously were not possible given the 
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structure of the data), there is a significant risk of 
impact analyses resulting in false positives (i.e., 
detecting false significant treatment effects) due to 
actual clustering effects that were not included in 
the statistical models. Given these risks, this power 
analysis should be interpreted cautiously. 


SRI explored interaction effects between the 


treatment condition and four student-level predictors: 


gender, race/ethnicity, Pell status, and enrollment 
status. These analyses were conducted using 
ordinary least squares regression. Given that 
exploring interaction effects decreases statistical 
power, datasets were only analyzed for interaction 
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effects if they contained at least 60 students in each 
of the treatment and control conditions. This sample 
size requirement was chosen out of convenience, 
allowing the sizable majority of datasets to be 
explored for interaction effects. The overwhelming 
majority of the interaction effects analyses contained 
identical predictors to the standard impact analyses 
to allow for meaningful comparisons across the 
models. For one institution the external prior 
achievement measure was removed from the 
interaction effect analysis to meet the increased 
sample size requirement. 
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Appendix D: Meta-analysis Methods 


A meta-analysis was used to synthesize the results 
from the final 28 impact analyses. The difference in 
means from the impact analysis, common standard 
deviation of the outcome, and sample size for the 
treatment and control groups from each impact 
analysis were entered into the Comprehensive 
Meta-Analysis version 3 (CMA) software program. 
Moderator information for a given institution and 
course was collected through an instructor survey 
and from other corresponding research design 
documents. The meta-analyses were run assuming 
a random effects model and reporting Hedge’s G as 
the effect size. The Q statistic for heterogeneity was 
computed to determine whether variation among 
the 28 impact estimates could be explained by 
circumstantial characteristics, such as course subject 
matter, class size, courseware modality, or institute 
type (4-year versus 2-year). 
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Appendix E: Cost and Cost Effectiveness Study Methods 


Cost information for a given institution and course 
was collected through an instructor survey, on- 

site visits, and follow-up communications with 
administrators at the respective institution. All 
relevant data were entered into a template that 
captured possible cost impacts for setting up and 
delivering courses with the NGC courseware being 
evaluated and BAU, both in the initial year and on an 
ongoing basis. We took an “ingredients” approach’, 
which identifies all inouts regardless of whether or not 
administrators, faculty members, or students bear 
the costs. The method is based on the work of Henry 
Levin and colleagues, who established a procedure 
for quantifying the opportunity costs and accounting 
for relevant costs (Levin 1975 & 2001).?,° These 
categories included the costs of providing instruction 
(staff professional development, 


1 An “ingredients” approach consists of 3 primary steps: 
(1) Selecting and defining relevant cost components (a.k.a., “ingredients”) 
(2) Collecting information on each ingredient and applying a monetary valuation 
(83) Calculating the total cost across all ingredients 


2 Levin, H. (2001). Waiting for Godot: cost effectiveness analysis in education. New directions for evaluation, 2001(90), 55-68. 


3 Levin, H. M. (1975). Cost-effectiveness analysis in evaluation research. Handbook of evaluation research, 2, 89-122. 
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SRI Education, a division of SRI International, is tackling the most complex 
issues in education to identify trends, understand outcomes, and guide policy 
and practice. We work with federal and state agencies, school districts, 
foundations, nonprofit organizations, and businesses to provide research-based 
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change. SRI International is a nonprofit research institute whose innovations 
have created new industries, extraordinary marketplace value, and lasting 
benefits to society. 
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